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Introduction 


A revised version of the IDEA form for collecting student ratings of instructional processes 
and outcomes has been administered since the fall term of the 1998-99 school year’. Results 
from all administrations of the device from August 1998, through August 2001, constitute 
the basic data of this report. A total of 122 institutions of higher education participated in 
the program during this time span; reports were prepared for 73,722 classes”, of which 
29,267 used the Short Form and 44,455 used the Diagnostic (long) Form. 


No claim is made that participants are representative of American higher education. 
However, they are relatively diverse, both geographically and in mission. Table 1 shows 
information about the highest degree offered by participating institutions as well as their 
geographic location. 


Table 1 
Number of Institutions Included in Research 


Highest meres Offered 
Soe I 
|East/Northeast | 7 | S| TS 
[Midwest | 8 | S| 
[Southwest | S| 3ST 
Rockies/West | 4 fT 


Fifty-five institutions were publicly supported, 44 were private not-for-profit, of which 
many were church related, and 23 were private for-profit. Enrollment varied widely from 
under 500 (11 institutions) to over 20,000 (9 institutions). The two most common size 
categories were 1000-2499 (28 institutions) and 5000-9999 (29 institutions). 


In terms of classes processed, 22 percent were from two-year institutions, 14 percent from 
those whose highest degree offered was the bachelor’s, 28 percent from Master’s degree 
institutions, 23 percent from doctoral institutions, and 13 percent from other types of 
institutions. 


This report is organized into six parts. 

I. Basic Data (including means, standard deviations, norms for types of institution, 
and inter-correlations of all items) 

II. The Structure of the Ratings 

Ill. The Process of Adjusting Ratings 

IV. Reliability 

V. Validity 

VI. Other Technical Questions 


' Copies of the instruments and sample copies of reports to participants are included in Appendix A. 

* Institutions that were first-time participants in the IDEA program were excluded, as were classes with fewer 
than 10 respondents. Furthermore, if a single institution contributed more than 5% of the classes processed in 
a given year, classes from that institution were randomly deleted until the remainder constituted only 5% of the 
total. 


Section I. Basic Data 


This section presents item means, standard deviations, and inter-correlations as well as 
percentile ranks for all institutions and for each of four types of institutions (defined by 
highest degree offered). The data are based on the 44,455 classes that employed the 
Diagnostic Form in the time period from August 1998, through August 2001. 


Table 2 describes faculty ratings of the importance of the 12 learning objectives as reported 
on the Faculty Information Form (FIF). A 3-point rating scale was used for these 12 items: 
“1=Of no more than minor importance;” “2=Important;” and “3=Essential.” The table 
shows the number of classes for which a given objective was identified as “important” or 
“essential,” the mean and standard deviation, and the percent of classes where the objective 
was identified as “essential” or “important.” 


Table 2 
Faculty Ratings of the Importance of Twelve Learning Objectives 


Learning Objective 
8 J (Important & 3 3 Mean? 
Essential) 


| Merentiaionsertendes | 998] | at | ate [80 
generalizations, or theories ° : 
3. Learning to apply course material (to 
improve thinking, problem solving, and 30,442 40 35 2.10 77 
ae ESmiGrie 
of view needed by professionals j ; ; 
pent | oes | | 8 | te [os 
member ; : : 
6. Developing creative capacities--writing, art jvm fete fie 
etc 


7. Gaining a broad understanding, 
appreciation of intellectual/cultural activity 10,256 17 
(music, science, etc.) 


8. Developing skill in expressing oneself 18,174 %6 0 1.67 19 
orally or in writing. 


| 9-Leaming how to find and use resources | 15,656__| 31] 0 [1367] 
10. Developing a clearer understanding of, and Pans [7 [6 | 130 [ow 
commitment to, personal values. 
Dene ee ee 
ideas 


12. Acquiring an interest in learning more 15616 | 30 | Il | 152) | 68 | 


“Percentages based on all classes employing the Diagnostic Form. Percentages will not equal 100 because the 
ercentage indicting the objective was “Of minor or no importance” are not reported. 
A 3-point rating scale was used: 1=Of no more than minor importance, 2=Important, 3=Essential. 


A review of Table 2 provides an indication of the instructional priorities of those 
participating in the IDEA program. The first four objectives are stressed most frequently; 
these represent the acquisition and application of basic cognitive background, often as a part 
of professional preparation. Academic skills (8. communication; 11. critical analysis) were 


also stressed frequently, but not as often as the first four objectives. Next in importance 
were the two “life-long learning” objectives (9. finding and using resources; 12. interest in 
learning more). The objectives that were stressed least were those concerned with values 
development (item 10), creative capacities (item 6), and a broad liberal education (item 7). 
American higher education is often portrayed as pragmatic and utilitarian; these results are 
consistent with that stereotype. 


Table 3 gives the mean, standard deviation, and number of classes for the 47 individual 
items rated by students. A 5-point rating scale was used throughout, with “1” representing 
the lowest rating (least frequent, least characteristic, least satisfactory) and “5” the highest 
rating. 


In addition, two “overall effectiveness” measures were included—PRO (Progress on 
Relevant Objectives) and PRO aq. PRO was derived by combining the faculty member’s 
ratings of “Importance” of a given objective with the average student rating of “Progress” on 
that objective. Because the average student rating of progress is different for each of the 12 
learning objectives, these averages were first expressed as T Scores, a mathematical way of 
converting all averages to 50 and all standard deviations to 10°. These T Scores were then 
weighted by the faculty member’s rating of the importance (relevance) of each objective. 
For objectives rated as “Essential,” the T Score was multiplied by 2 before being added to 
the T Score for objectives chosen as “Important;” objectives rated as “Of no more than 
minor importance” were ignored. The PRO measure was derived by dividing the sum of the 
weighted T Scores by the sum of the weights. The PRO qj measure adjusts PRO by taking 
into account factors which influence student ratings but which are beyond the control of the 
instructor. The adjustment process is described in Section III of this report. 


For the student ratings shown in Table 3, it should be noted that, although “3” was the 
midpoint of the rating scale, all ratings averaged above “3” and 13 of them averaged above 
“4.” While these relatively high ratings probably reflect a generally high quality of 
instruction being provided at participating institutions, they are also due in part to a tendency 
for students to be “lenient” in their ratings. This is revealed most clearly in those items 
where students are asked to compare the class with others they have taken (Items 33-35), 
where averages were 3.20, 3.42, and 3.42, respectively—well above the average which 
would be expected if leniency were not an issue. 


; T=50+[10(X-M)/SD] where X=mean for the instructor; M=mean for the comparison group; SD=standard 
deviation for the comparison group. 


Table 3 
Student Ratings of Individual Items on the IDEA Diagnostic Form 
Student Ratings of Teaching Methods Mean 


1. Displayed a personal interest in students and their learning. 44,451 : 
2. Found ways to help students answer their own questions. 44,448 : 


3. Scheduled course work in ways which encouraged students to stay up-to-date in 44.447 4.20 ‘48 
their work. 


5. Formed “teams” or “discussion groups” to facilitate learning. 1.03 
| 14. Involved students in “hands on” projects (research, etc.) | 44,443 | 3.76 | 80 | 


16. Asked students to share ideas and experiences with others with different 44,445 


backgrounds and viewpoints. 


19. Gave projects, tests, etc. that required original thinking. 
| 44. Used a variety of methods to evaluate student progress. 44,442 | 3.83 | 60 | 
45. Expected students to take their share of responsibility for learning. 
46. Had high achievement standards in this class. 


Student Ratings of Progress 
21. Gaining factual knowledge (terminolog 
22. Learning fundamental principles, generalizations, or theories 
23. Learning to apply course material (to improve thinking, problem solving, and 
decisions) 
24. Developing skills, competencies, and points of view needed by professionals in 44.44] 3.9] 54 
the field most closely related to this course , 


25. Acquiring skills in working with others as a team member 44,437 
26. Developing creative capacities (writing, inventing, etc.) 44,438 


27. Gaining a broader understanding and appreciation of intellectual/cultural activity 44.440 | 3.32 74 
(music, science, literature, etc.) : : : 


| 44,439 | 3.41 | 80 | 
| .69 | 


| 44,437 | 3.74 | 56 | 
3 
3 


Ratings of Course Characteristics 


: 4 
44,43 : : 
33. Amount of reading 44,447 
34. Amount of work in other (non-reading) assignments 44,445 
4,44 : 
4,44 : 
: ee) 
; A 
: 3 


35. Difficulty of subject matter | 44,445 | 3.42 | 58 | 


Self-Ratings 


Table 3 is continued on the next page. 


Table 3 (continued) 
Student Ratings of Individual Items on the IDEA Diagnostic Form 
Global Ratings of Outcomes 
40. As a result of taking this course, I have more positive feelings toward this field 
of study. 


PRO-Adjusted 
*PRO ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10. All 
other ratings were made on a 5-point scale where 1 is low and 5 is high. 


Inter-correlations for all items included in Tables 2 and 3 are provided in Tables 4, 5, and 6. 
Refer to Tables 2 and 3 for item descriptions. 


The correlations shown in these tables may seem overwhelming. Aside from their value as 
basic information, they can help the reader gain a deeper understanding of individual ratings. 
For example, there may be interest in understanding factors that relate to how hard students 
work in a class (Item 37: “I worked harder on this course than on most courses I have 
taken’). As shown in Table 6, although a substantial number of items were significantly 
correlated with responses to this item, the highest correlations were with items related to the 
instructor’s course management and/or expectations. Thus, means on this item correlated 
.68 with the amount of other (non-reading) work assigned in the course (Item 34), .67 with 
the difficulty of the course (Item 35), .66 with the instructor’s achievement standards (Item 
46), and .54 with the instructor’s tendency to hold students responsible for their own 
learning (Item 45). Similarly, the perceived difficulty of a course (Item 35) was largely a 
function of the magnitude of assignments given (reading, Item 33; other, Item 34) as well as 
the instructor’s achievement standards (Item 46) and success in stimulating student effort 
(Item 8). Detailed analyses such as these can result in new insights regarding teaching, 
learning, and the IDEA system. 


Table 4 
Inter Correlations of IDEA Faculty Information Form 
Faculty Ratings (FR) 

FRI [io, | | | | | | | | | | | 
ERO AOA R0O re ee — 
Fo A O72 a (CR) (| 
Fre [3 | to, jit | | | | | | |_| 
PrRS_[-03| 04 | 27 | 2% [io> [| [| | | | | 
PR6[-11[-o[ 3[ 2| 29 fio, {| | | | |_| 
|FR7_ | -.04 | -.01 | -.03 | -.04 | 12 | 33 | 100{ | ea 
PrRE_|-22[-14[ 06 [ or 31] 34 [24 [100] 

Prro_| 07 | 10] 22 | 3 | 34) 28 | 17) 3810; | | | 
PFRIO|-00[ 08 [21 [ 10 | 29 [22 | | 2) 32 fio] | | 
FRI [- | 07] 23] ol] 2), ma] a[ ai, 38 [iol] | 


See Table 2 for item descriptions. 


Table 5 
Inter Correlations of IDEA Faculty Information Form (FR) 
and IDEA Diagnostic Form (SR) 

rsrt_[-07 | -06 | 00 | 05 | 04 [05 [00 | 04 [01 | 07 | 00 | 03 | 
SR? [-08 [-06 [03 [05 [oF | |---|] [__ 
|SR3_ | -.03 | -.05 | 02 | .04 | .00 | .02 | -.03 | .03 | -01 | .00_| -.03 | -.01 | 
pea [02 for [ot 06 [00 for [03 [02 [=o [09 | 02 [0 
[srs [24 [18 {06 [06 | Ge Ga a a 
see [or [03 [or Fo5 [02 | -oz [or [or [=o | o7_| 02 [00 | 
SRT ats | 1201 | 09] 09-16 | 0214 08-0605 [03] 
PSR8_|-05 [-03 | 03 [05 [03 [04 [00 [05 [02 |_04 | 06 [03 | 
SRO [-14[-4[ [7,2] 10 [-0| a, 2] 0% | 2 | 06 | 
-SRIO[ 00 | -03 | -.03 [02 | -03 | 00 [01 [02 [-03 [04 | -01 _|_.00_] 
rsri_| 02 [02 [07 [07 | 06 [-07[-10[-m | 0 | 4 | m2 | 03 | 
PsRI2 [13 [07 | 02 [01 | -06 | 10 | -.06 | -.11 | -06 | -.02 | -09 | -03 | 
SRIB [04 | -05[-02 | 2] 3] | 7 | ml O13 | 06 | 06 | 
srig[-12[-13 | 10 | 23 | 25 | 13 [-08 | 08 [15 | 07 | 00 | 04 | 

SRIS | -12 [-10 [06 | 15 | 13 | 14 [-03 [08 [08 |_09 [02 [05 
PsRI6[-2[-17, O| Bl 7,2, ol up ol 3] 9] 12 | 
sRI7 [01 | .00-[ 00 [01] -02 | -02 [03 [02 [-03 [00 | -02 | -07 | 
|SRI8 | -17 | -.13 | 05 | 10 | 20 | 09 | -.02 | 12 | 05 | 10 | 05 | .05_| 
SRO -24[-18[ 3 | O] 4, a] oe, 0] |_or| 
-SR20[.06 [=05 | 01 | .03-|_04 | 02 [-.06 [03 [02 [00 | o1_| -or | 
PSRa_[ 2a [A [04 | 12 | -05 | -09 | -10 | -17 [05 | -05_| 11 | -02 | 
PSR22 [14 [a7 | 09 [11 | -02 | -07 |-13 | -17 | -06 | 01 | -07 | _00_] 
| SR23 | -.04 [ -.01 | 14 | 19 | 07 | 03 | -.16 | 03 | 02 | 04 | -.04 | 01 
|SR24 | .00 | -.03 | 08 | .26 | 08 | 07 | -.14 | -.04 | 02 | -.00 | -.08 | .00_| 
| SR25 | -.18 | -.14 | 10 | 15 [| 39 [ 08 | -.07 [ 14 | 09 [| 08 | 02 | 04 | 
| SR26 | -.32 | -.27 | -.04 | 10 | 17 [37 [| 17 [35 | 12 | tt | 16 | 09 | 
| SR27 | -.18 | -.18 | -.11 | -.02 | 08 | 25 | 33 | 22 | 05 | 14 | 14 | 1 | 
| SR28 | -.32 | -.26 | -.04 | 01 | 17 | 19 | 12 | 46 | 13 | 16 | 24 | 09 
| SR29 | -.10 | -.10 | 08 | 12 | 12 | 05 | -.09 | 16 | 21 | 02 | 08 | .05_| 
SRI0[-d6 | -d1 | 03 | 05 | 13 [08 | 02 | 5 | 0s | 28 | 15 | | 
|SR31 | -.21 | -.12 | .02 | -.02 | 08 | 08 | .03 | 23 | 07 | .16 | .27 | .08 | 
SR32[-09 | -.06 | 05 | 10 | 08 | 07 [-02| .06 | 06 | 11 | 08 | 09 
| SR33_[ 01 | 01 | -.04 | -.13 | -05 | -.18 | 08 | 13 | 00 | 06 | 21 | 03 | 
SR34[-06 | -05 [12 [19 | 08 | 12 [-12 | 06 [07 | -13 | -06 | -05 | 
PSR35[ 16 | 17 | 05 | 02 | -12 | -11 | -08 | -.16 | -08 | 18 | -05 | -07 | 
PSR36 [08 [03 [03 | 26 [07 | 11 | -04 [-11 [-02 [05 | -10 |_05_| 
PSR37 [04 [03 | 07 | 16 | Or | .06 | -10 | --02 [00 | -10 | -.04 | -03 | 
PSR38 | -O1 | -03 [01 | 13 [04 [04 | -04 | -06 [-03 [2 _| -07 | -01_| 
| SR39 | 08 | 04 | 06 | 25 | 09 | 10 | -.05 | -.09 | O01 | 03 | -.10 | .05_| 
rsra0|04 [01 [02 | 18 | 05 [07 | -02 | -06 [-02 [08 | -06 [04 | 
SRT | 05 [-05 | -03 [00 | -01r [02 [01 [02 [-03 |_04 | 00 | 00 | 
| SR42 | .00 | -.03 | -.01 | 11 [| 03 | 08 | .00 | -.01 | -.03 | .07 | -.04 | .03_| 
rsRas [00] -02 [07 | 17 [09 [05 | -03 |-05 | 02 [01 | -o4 |__| 
rsrag[-12[-12 | 08 | 15 | 16 | 09 [-03| 12 [07 |_05_| -o1 | 02 | 
rSRa5 | -04 | -06 [01 | 10 | 04 | 03 [-03 | or [-or|_o1_| -o1 | 0 _| 
rsr46[ -.03 | -05 | 02 | 10 | 02 | 05 [-04 [04 [-o1 [01 | 00 | -02 | 
rsra7 [00 [-07 | 07 [14 [09 [01 [10 [00 [14 | -07 | -05 | -01 | 

Bold numbers are correlations between student (SR21-SR32) and faculty ratings (FR1-FR12) of the twelve learning 

objectives. 

See Tables 2 and 3 for item descriptions. 
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Table 6 
Inter-Correlations of IDEA Student Ratings (SR) — Diagnostic Form 


Petia = | 
Poet) eave IN MAO ttt ps Ss IS ee Ee a a a = ee 
FSS SS A eH ee HD (ie (O(n (ne a A] A ee | ee (ee 
|SR5_| 41 | 44 [36] 33{ 1o{ | {| | {[ | | | fT ft | fT | | ft of ft 
DSSS OR a RB a a F(R Fa a a (es (fee ey | ee ee 
Fea OP ee dW (A (A | (A (ae ee 
DSS Da ed (Fe (| ee ee ee) 
[SRO | 54 | 56 | 49] 54{ 48 | 53 | oo] or} rot | | ot tt 
| SRIO | 77 | 81 | 76 | 83] 27 | 86 | 71 | 69] 48] io | | | 
FS A a ee ee ee ee ee ee ee eee ee 
HESS ee We DD (P| a a eS ee ee ee 
| SRI3 | 78 | 82 | 69 | 86 | 40 | 86 | 74 | 79 | 59 | 81 | 79 | o8 | 10 [ {| [| ft ft of ft 
FS A 2 DP HD Oe (ee ee ee ee eee ee 
PORTS 7h | S28 Ie 360% ee 1S | 5 Le] SA 282 8 | 2972 x99 5 262.4) 250.2) 279) 70" | Oe fe 
BSRTO: 63 | 1965) AOU] 2595/1 364.3| 264.66. 607 | 15655]. 253.3), G43 | 24 | oA 70 AO) 2) | fs Ts ft 
| SRI7 | 66 | 67 | 71 | 64 | 26 | 66 | 65 | 61 | 41 | 70 | 52 | 68 | 62 | 35 | 59 [ 45 | 10 | [| | fT 
SETS 278 | 2296. | OL O18] 272s) OA cI 7S | 268 <| SB 2] STs SA AB le sOT | 05 | 5 STI OSI a ff 
| SRI9 | 61 | 65 | 59 | 58 | 56 | 59 | 70 | 66 | 68 | 54 | 50 | 45 | 68 | o9 | 74 [| 74 {| 48 | op | io] | | | 
ESR20 | 74 | 70 | sGL | 362] 38] 635 567 [ 68 | 2556)! 59353] 54. | 64 | AT | 69 [53 | 5864 | 56 1 
| SR21 | 60 | 66 | 62 | 72] 18 | 73 | 57 | 72 | 42 | 68 | 59 | 69 | 68 | 40 | 63 | 36 | 57 | 48 | 40 | 55 [| 10] | | 
fe SR22 IN G4, |568-] G2N 592. | 22) AY 59. 78 | AL ot | 60] 367 69 | AT |c05" [LA 57 i520 ie AA 55) BO Oe | 
BROS fh 70] FF | 368 |) 76) AO 274 70176. 53 || 269.68" ||| 63: | 274 | 6078] 57 | 59: || 267°] 62.) 61 |) 76] 80] 107 | | 
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Table 6 (continued) 
InterCorrelations of IDEA Student Ratings (SR) — Diagnostic Form 
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See Table 3 for item descriptions. 


Of special interest is the relationship between ratings of teaching methods and instructional 
outcomes. Are some teaching approaches more closely associated with progress of a given 
type than others? Do the most effective methods differ depending on instructor objectives? 
Answers to these questions are highly relevant to the IDEA system’s goal of facilitating 
instructional improvement. 


Although a review of relevant correlations in Tables 4, 5, and 6 provides a direct approach 
to this problem, it is commonly assumed that answers may depend, in part, on class size. 
Therefore, correlations between instructional methods and student ratings of progress were 
computed separately for four class sizes—small (10-14), medium (15-34), large (35-49), and 
very large (50+). Table 7 shows the “methods” items, which were most closely related to 
progress ratings on each objective for each of these four class sizes. Typically, seven to ten 
methods were identified as “most” closely related to progress ratings. 


Although there was some overlap between the lists of “most relevant” items (especially 
between the first two objectives), the pattern of items tended to be distinctive for each 
objective. Differences among class sizes were not dramatic, but were large enough to merit 
a separate listing of “most relevant items” for each size group. 
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Table 7 
Relationship of Teaching Methods to Learning Objectives (Correlations) 


| ta rte 
Factual Knowledge Theories 
a) ee 
| 1.DisplayedpsnlinterestinSs [| | | | 
| 2. Helped SsanswownQs | 65 | 69 | 69 | 66 | | _.68 | 71 | 73 | 75 _| 
[3. Scheduled workhelpfully [64] | | | [ | | | | 
| 69 | 72 | 72 | 73 | 
| 5.Formedteams, discussion [| | | | | =| ot | 
| 7.Explained criticisms | | | | | FL 
| 9.Energd multipleresources | {| | | | | 
| 11.Relatedtoreallife | 4 | | | 
12. Tests cover impr. pomis | 68 | 09 | 10 | 0 

Pea 


14. Involved Ss in “hands on” 


| 69 | 
| 69 | 
| 
peer 
eal 
| 69 | 
| 69 | 
eat 
Ei 
| 
| 
= 
| 
= 


Ny Ny yyN Nya] Ny Simi 
\o ~— oO}e CO} | 00 \o CO} 


Na ~ Naf} Na N tC 
& oO \O] KW] ON Co ~ 

Na NafNajr Nn N < 
= CO] Uo[ Nn nN nn lee 


© 
S 
a 


is =] 
eo ete ae, 

Obj. 24. Prof. Skills, j. 26. Creative 
aa ete ae tI MT | VE | 
TT Displayed palnierestinSs | 6770] |_| 
| 3. Scheduled work helpfully [| | | | 
FS. Formed teams, discussion | [| |_| 
| 7.Explained criticisms | 68 | 72 | .73 | .73_| 

8. Stimulated intellectual effort 
9. Encrgd multiple resources z= a ia 
10. Explained clearly 

11. Related to real life 

12. Tests cover imprt. points 


aaa 
13. Introduce stimulating ideas | .69 | 
| 


| 14.InvolvedSsin“handson” | | |_| 
| 15. Inspiredtosethigh goals | .76 | 78 | 80 | .79 | 
| 16. Asked to share experiences | | [| | 
| 17.Providedtimely feedback [| [| [| =| | |__| i 
18. Asked Ss tohelpeach other | | | | 68 | 
| 19.Creative assessments | | | | | |_| 53 | 56 | 8 
| 20. Enrgdoutclass S/Fcontact | | | | | | | | J | 
S=small (10-14), M=medium (15-34), L=large (35-49), VL=very large (50+) 
Only the most highly correlated items are shown. 
Note: Analyses reported in Table 7 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


SY 


Nalin wn nln in] oar nl nla 
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Nn na in nla wn < 
love] ie) Ne oo an ~) 
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Table 7 is continued on the next page. 
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Table 7 (continued) 
Relationship of Teaching Methods to Learning Objectives (Correlations) 


en 
Liberal Education cation Skills Resources 
ee ee SS 

| 1. Displayed psnlinteestinSs |.50] | | 

| 3. Scheduled workhelpfully | | =| | 

| 4. Demonstratedimp of subject | |_| 57 | 52 | 

| 5.Formedteams, discussion | | | | 

a a a) Se (dee ae 
| 62 | 65 | 62 | 66 | 
| 8. Stimulated intellectual effort | 50 | .60 | 59 | | | 726 | 667] 
| 9.Encrgd multiple resources | | | | 
10. Explained clealy | | 58 | 60 | 31] 

| 11.Relatedtoreallife | S| S| | 

| 12.Testscoverimprt. points | | | | 

| 14.InvolvedSsin“handson” | | | | | | | [| | | [63 | 64] 69 | 73 | 
| 16. Asked to share experiences |__| 57 _| 60 | 59 | 
| 17.Providedtimely feedback | | | | 

| 18.AskedSstohelpeachother| | | | : 

| 20. Enrgd out class S/Fcontact | [| | | | | | [ | | 


Development. Analysis Learning 
|S | M | 
| 1. Displayed psnlinterestinSs_| 61 | | 69 | 63 | 
|_| 


pe ol ee ee ee 
65 | 
| 5.Formedteams, discussion [| | | | | | | | | | | [| [ | 
(6. Made clear how topics ft__| 61 | 69 [73 | 65 | a| ope ar_ 
| 7.Explained criticisms | | | | CL 
| 9.Encrgd multiple resources [| | | | 
"10. Explainedclearly |__| | |_| 
| 11.Relatedtoreallife | 64 | 71 | 67 [ 
| 12.Testscoverimprt.points | | | | 
| 13. Introduce stimulating ideas_| .70 | .77 | 78 | .69 | 
| 14. Involved Ssin“handson” | | | | 
15. Inspiredto ethigh goals | 66 [71 | 09 | 61 | 
| 17.Providedtimely feedback | | | | 
| 18. Asked Sstohelpeachother | 66 |.69 | | 
19. Creative assessments | | | | 
| 20.Enrgd out class S/Fcontact |_| | | 

S=small (10-14), M=medium (15-34), L=large (35-49), VL=very large (50+) 

Only the most highly correlated items are shown. 

Note: Analyses reported in Table 7 used a more restricted data set. Classes with response rates less than 75% 

or not reporting the number enrolled were also excluded. 
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Class size is relevant in another way. Average ratings of the frequency with which each 
method is employed varies with the size of the class. These ratings also vary with the 
degree to which students were motivated (really wanted the course regardless of who taught 
it). Faculty members participating in the program want to know if their ratings were above 
or below average, especially on those items shown to be most related to progress on 
objectives they have chosen. 


To obtain a meaningful answer to this question, it is necessary to know the average rating 
for each item for classes grouped according to both class size and student motivation. 
Accordingly, four class sizes were identified: Small (10-14), Medium (15-34), Large (35- 
49), and Very Large (50 or more). Similarly, five “motivation” levels were established, 
representing roughly the upper 10 percent (High), the next 20 percent (High Average), the 
middle 40 percent (Average), the next 20 percent (Low Average), and the lowest 10 percent 
(Low). By jointly considering these two classification methods, a 4 x 5 table was 
constructed consisting of 20 cells (one for each combination of class size and student 
motivation). Average scores on each of the 20 teaching methods items were then computed 
for each item. Results are shown below in Table 8. 


Table 8 
Average Scores for Method Items by Class Size and Level of Student Motivation 


1. Displayed a personal interest in students and their learning 


ow 


2. Found ways to help students answer their own questions 


3. Scheduled course work (class activities, tests, projects) in ways which encouraged 
students to stay up-to-date in their work 


P| gg ee linens 


= 
oy 
(oe) 
+ 
Nas7) 
S 
ie) 
os) 
iss) 
> 
oS 
S 


— 
S 
oO 

no} 
f= 
nN 


otivation (#39) 


High Average 


Motivation (#39) 


Table 8 is continued on the next page. 
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Table 8 (continued) 
Average Scores for Method Items by Class Size and Level of Student Motivation 


4. Demonstrated the importance and significance of subject matter 


ae | 


5. Formed teams or discussion groups to facilitate learning 


6 — aa ee ee ee 


6. Made it clear how each topic fit into the course 


7. Explained the reasons for criticisms of students’ academic performance 


eee ee 


8. Stimulated students to intellectual effort beyond that required by most classes 


= 
nN 
foe) 
+ 
ie 
Q§ 
oes 
BS 
nN > 
oa 
= 


Student 
Motivation (#39) 


Student 
Motivation (#39) 


Student 
Motivation (#39) 


High Average 


Student 
Motivation (#39) 


Table 8 is continued on the next page. 


14 


Table 8 (continued) 
Average Scores for Method Items by Class Size and Level of Student Motivation 


9. Encouraged students to use multiple resources...to improve understanding 


CC ae oe ee 


10. Explained course material clearly and concisely 


Ld ee ee ee 


11. Related course material to real life situations 
ow 


12.Gave tests, projects, etc. that covered the most important points of the course 


| a ee oe eer 


13. Introduced stimulating ideas about the subject 


Class Size (Enrollment) 
Very Large 


PY 
nN 
lag) 
Fk 
— 
i= 
ie) 
y=! 
i>) 
> 
an! 
S 


otivation (#39) 


Student 
otivation (#39) 


Motivation (#39) 


High Average 


Motivation (#39) 


'W 


Table 8 is continued on the next page. 
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Table 8 (continued) 
Average Scores for Method Items by Class Size and Level of Student Motivation 


14. Involved students in hands on projects such as research, case studies, or real life 
activities 
i oe ee ee ee oe 


15. Inspired students to set and achieve goals which really challenged them 


Motivation (#39) 


High 


Student 
Motivation (#39) 


16. Asked students to share ideas and experiences with others whose backgrounds and 
viewpoints differ from their own 


a ee ee eo 


17. Provided timely and frequent feedback on tests, reports, projects, etc. to help students 
improve 
a ee eae es ee eee 


Student 
Motivation (#39) 


Student 


= 
oy 
(oe) 
+ 
Nas) 
S 
° 
oS 
isc) 
> 
3S 
S 


Table 8 is continued on the next page. 
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Table 8 (continued) 
Average Scores for Method Items by Class Size and Level of Student Motivation 


18. Asked students to help each other understand ideas and concepts 


Class Size (Enrollment) 
Very Large 


19. Gave projects, tests, or assignments that required original or creative thinking 


Class Size (Enrollment) 
Very Large 


Motivation (#39) 


Motivation (#39) 


20. Encouraged student-faculty interaction outside of class (office visits, phone calls, email, 


etc. ) 
= SS Lee 
E 
z= 
g 


Note: Analyses reported in Table 8 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


The information provided in these cells is intended to provide diagnostic assistance to those 
using the Diagnostic Form (see pages 4 and 5 of the sample IDEA Report included in 
Appendix A). This is done through a series of steps. 


First, “relevant” objectives are identified (those the instructor identified as “Important” or 
“Essential”). Then, the most relevant teaching methods—those most closely related to a 
given progress rating—are identified (see Table 7). The class is then classified according by 
its size and level of student motivation. Results on the “most relevant” items are then 
compared with those for “similar classes” using the data reported above. 


If the obtained mean is 0.3 (approximately one standard error) or more above the mean for 
similar classes, the user is encouraged to retain this approach; if it is 0.3 or more below the 
mean for similar classes, the user is advised to “consider increasing the frequency” with 
which the method is employed. 


Table 9 provides normative information for each of the items included on the Diagnostic 
Form. Separate norms for the Short Form are not included for reasons described in Section 
VI of this report. 


il 


Norms are provided for all institutions and for those whose highest degree offered is the 
Associate (2-year), Baccalaureate, Master’s, or Doctoral. As noted earlier, a number of 
“Other” institutions also participated. These were principally institutions with highly 
specialized emphases; they were so heterogeneous that a meaningful norm (comparison) 
group could not be described. 


For items or measures that are intended to provide information about the effectiveness of 
instruction, norms are provided for both unadjusted (raw) and adjusted scores. Of these, 
Items 21-32 represent student ratings of the progress they made on each of 12 learning 
objectives; for these 12 items, the only classes included are those for which the objective 
was rated as “Essential” or “Important” by the instructor. The process of adjusting scores is 
described in Section II of this report. 


Table 9 also provides norms for five “scales” descriptive of alternative teaching approaches 
or styles contained in the IDEA Survey. A further description of these scales is provided in 
Section II of this report. 


As shown in Table 9, for the most part, differences among types of institutions were 
relatively slight. There appeared to be a tendency for ratings to be slightly higher at two- 
year institutions. For example, on Item 17 (frequency and timeliness of feedback) an 
average of 4.3 was at the 49'" percentile for 2-year colleges but at the 61‘ percentile for 
those offering the baccalaureate degree. Similarly, on Item 47 (use of educational 
technology), an average rating of 3.7 was equivalent to the 46"" percentile for 2-year colleges 
but the 57" percentile for 4-year colleges. But there were numerous exceptions The 
average ratings for the four types of institutions, given at the bottom of each table, were very 
close to each other. 


Differences among types of institutions were so slight that the IDEA Center will continue to 
use the al-classes norm in its reports. Users who feel more comfortable in interpreting 
results if they are compared with those from similarly classified institutions will find the 
necessary information in the Table 9 below. 


Table 9 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 
By Type of Institution 

1. Displayed personal interest 2. Helped students answer own questions 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS Doct. 
15 0 0 0 0 0 15 0 0 0 0 0 
2.0 0 0 0 0 0 2.0 0 0 0 0 0 
2.5 0 0 0 0 0 2.5 0 0 1 0 1 
2.8 1 1 1 1 1 2.8 2 1 2 2 2 
3.0 2 1 1 1 2 3.0 3 2 4 3 4 
3:3 4 3 3 4 5 3.3 7 5 8 8 9 
3.5 6 re) 6 6 8 35 12 9 13 13 15 
3.7 11 9 10 10 13 3.7 19 15 20 21 22 
39 17 15 16 17 20 39 30 25 32 32 33 
41 26 23 25 26 28 41 43 37 46 46 46 
43 38 35 37 38 41 43 59 = 53 62 62 61 
45 54 52 54 55 56 45 76 73 80 79 77 
47 74 73 73 75 74 47 90 89 92 91 90 
49 92 92 92 93 92 49 98 97 98 98 98 
5.0 98 98 98 98 97 50 99 99 99 99 99 
Avg. 43 44 43 43 43 Avg. 41 42 4.1 41 41 


Table 9 is continued on the next page. 
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3. Scheduled work helpfully 


Percentile Ranks for IDEA Diagnostic Form Items and Scales 


Mean All 2-Yr 


1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


Mean 
1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
4.5 
4.7 
4.9 
5.0 

Avg. 


5. Formed “teams” 


All 2-Yr 
2 2 
10 9 
20 20 
27-28 
31-33 
38 = 40 
43, 47 
49 53 
55 59 
62 66 
70 = =74 
79 = 82 
8 691 
96 «97 
99 99 
3.5. 39 


7. Explained criticisms 


Mean All 2-Yr 


1.5 
2.0 
2.5 
2.8 
3.0 
33 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


0 


3.8 


0 


3.8 


Table 9 (continued) 


By Type of Institution 


BA,BS MA,MS Doct. 


0 0 
0 0 
0 0 
1 1 
2 2 
5 5 
9 9 
15 15 
24 24 
a9 37 
53 54. 
2B 73 
89 89 
98 98 
99 99 
42 42 
BA,BS MA,MS 
3 3 
ul in 
21 22 
29 28 
34 32 
4 39 
46 44 
52 49 
58 56 
65 62 
2B 70 
81 79 
90 87 
97 96 
99 99 
34 35 
BA,BS MA,MS 
0 0 
0 0 
2 2 
4 5 
8 9 
16 20 
26 31 
38 44 
52 59 
67 ie 
80 84 
90 92 
96 97 
99 99 
99 99 
38 37 


Table 9 is continued on the next page 
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4. Demonstrated significance 


Mean All 2-Yr 


Is 
2.0 
25 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


All 2-Yr 
0 0 

0 0 

0 0 

1 1 

2 2 

5 5 

9 9 
15 14 
23. 23 
34. 34 
50 49 
68 ~=—«68 
85 85 
97 97 
99 99 
42 42 


BA,BS MA,MS Doct. 


BA,BS MA,MS Doct. 


0 
0 
0 
1 
2 
) 
8 
14 
22 
32 
48 
67 
86 
97 
99 


42 


8. Stimulated intellectual effort 


Mean All 2-Yr 


15 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


BA,BS MA,MS 
0 0 
0 0 
3 1 
7 4 

il 7 
20 16 
29 25 
42 37 
56 50 
68 64 
79 77 
89 88 
96 95 
99 99 
99 99 
3.8 3.8 


9. Encouraged using multiple resources 


Mean All 2-Yr BA,BS MA,MS Doct. 


15 
2.0 
2D 
28 
3.0 
3:3 
3.5: 
37 
3.9 
41 
43 
45 
47 
49 
5.0 


Avg. 


Mean All 2-Yr 


Mean All 2-Yr 


15 
2.0 
2.5 
28 
3.0 
3.3 
3.5: 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 
Table 9 is continued on the next page 


Table 9 (continued) 


Percentile Ranks for IDEA Diagnostic Form Items and Scales 


0 0 0 0 

1 0 2 1 
5 os) 7 6 
9 6 13 12 
14 +10 19 17 
23. «18 29 27 
31 = 26 37 36 
40 36 46 45 
S147 57 55 
61 58 68 66 
73 =«671 80 76 
84 83 89 86 
93 «92 95 94 
98 = =98 98 98 
99 99 99 99 
3.8 3.9 3.7 3.7 


11. Related to real life 


BA,BS MA,MS 

0 0 0 0 
0 0 0 0 
1 1 1 1 
2 2 3 2 
4 4 4 3 
8 8 9 7 
12 13 14 11 
18 19 20 17 
29: 27 29 24 
34-36 39 33 
46 49 51 45 
61 63 64 60 
77 ~=79 78 77 
93 94 93 93 
98 98 98 98 
42 42 42 42 


13. Introduced stimulating ideas 


0 0 0 0 
0 0 0 0 
1 1 1 1 
3 3 4 3 
5 5 6 5 
11 10 12 il 
17 15 18 17 
25 22 26 26 
3533 37 37 
48 45 50 50 
62 = 60 64 64 
77 ~=76 79 79 
89 889 91 90 
97 = 97 98 98 
99 99 99 99 
40 41 4.0 4.0 


BA,BS MA,MS Doct. 


By Type of Institution 


20 


Mean All 2-Yr 


10. Explained clearly 


0 15 0 0 0 0 
1 2.0 0 0 0 0 
5 2.5 2. 1 2 2 
10 28 4 2 4 4 
15 3.0 5 3 6 6 
24 33 ©6110 7 11 11 
32 35 14 10 15 16 
41 3.7 20 15 21 22 
51 39 28 22 29 31 
60 41 38 31 40 41 
72 43 52 43 55 55 
83 45 68 60 72 71 
92 47 84 80 88 87 
98 49 96 95 98 97 
99 50 99 98 99 99 
3.8 Avg. 41 42 4.1 4.1 
12. Tests covered important points 
Mean All 2-Yr BA,BS MA,MS 
0 15 0 0 0 0 
0 2.0 0 0 0 0 
1 2.5 0 0 0 0 
2 2.8 1 1 1 1 
4 3.0 2 1 2 2 
9 3.3 4 3 5 4 
13 35 7 5 8 7 
19 3.7 11 9 13 il 
27 39 19 15 21 19 
36 41 28 23 31 29 
48 43 42 36 46 43 
62 45 60 53 65 62 
77 47 80 75 84 82 
93 49 95 94 97 96 
98 50 99 98 99 99 
4.2 Avg. 43 44 4.2 43 
14. Involved in “hands on” 
Mean All 2-Yr BA,BS MA,MS 
0 15 0 0 0 0 
0 2.0 3 2 2 3 
2 2.5 8 8 7 10 
4 28 13 13 12 14 
7 30 618 ~— 18 16 19 
14 337° 25. ~27 24 27 
20 35 32 34 32 33 
28 37 40 42 40 41 
39 39 49 52 50 50 
50 41 59 62 60 60 
63 43, 69 73 71 70 
76 45 80 84 81 81 
88 47 90 92 91 90 
97 49 97 98 97 97 
99 50 99 99 99 99 
4.0 Avg. 3.7 3.7 3.8 3.7 


BA,BS MA,MS Doct. 


aAnNnNoOo 


Percentile Ranks for IDEA Diagnostic Form Items and Scales 


15. Inspired ambitious goals 


Mean All 2-Yr 


15 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


0 


0 


Table 9 (continued) 


By Type of Institution 


BA,BS MA,MS Doct. 


0 


17. Timely feedback 


Mean All 
1.5 0 
2.0 0 
2.5 1 
2.8 3 
3.0 5 
3.3 10 
3.5 14 
3.7 21 
3.9 29 
41 40 
43 54 
45 71 
47 86 
49 97 
5.0 99 

Avg. 4.1 


1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


2- Yr 


NDNWnNnNOoOcCO 


10 
16 
24 
34 
49 
67 
84 
96 
99 
42 


BA,BS 


BRNOO 


0 


19. Required originality 
Mean All 2-Yr 


0 


3.9 


0 


40 


BA,BS 
0 


3.9 


MA,MS 
0 


3.9 
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16. Asked diverse students to share ideas 


Mean All 2-Yr 


15 
2.0 
25 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


20. Encouraged out-of-class contact 


0 
2 
9 


18. 


0 
2 
7 
13 
18 
27 
35 
43 
54 
64 
75 
85 
93 
98 
99 
3.7 


BA,BS MA,MS Doct. 


0 


0 


0 
3 
10 
15 
20 
28 
34 
42 
50 
60 
70 
80 
90 
97 
99 
3.7 


Asked students to help others 
BA,BS MA,MS Doct. 


2- Yr 
0 


Mean All 2-Yr 


ie) 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


0 


0 


0 


BA,BS MA,MS Doct. 


0 


0 


n WN ee 
RESRGACUNScS 


21. Factual knowledge (unadjusted) 


Mean 
1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
4.7 
49 
5.0 

Avg. 


22. Principles, theories (unadjusted) 
Mean 


1.5 
2.0 
2.5 
2.8 
3.0 


23. Applications (unadjusted) 
Mean 


1.5 
2.0 
25 
2.8 
3.0 
3.3 
3.5 
34 
3.9 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 


All 2-Yr BA,BS MA,MS Doct. 


All 


0 
0 
0 
1 
3 


7 
12 
20 
34 
49 
68 
84 
94 
99 
99 

4.0 


2-Yr BA,BS MA,MS 


0 


3 
6 
14 
22 
33 
47 
63 
78 
89 
96 
99 
99 
3.9 


All 2-Yr BA,BS MA,MS 


0 


2 
4 
10 
16 
26 
39 
54 
69 
84 
93 
98 
99 
40 


0 


3 
5 
11 
20 
30 
44 
57 
71 
85 
94 
98 
99 
3.9 


21. Factual knowledge (adjusted) 


Mean All 2-Yr 


By Type of Institution 
0 0 1.5 
0 0 2.0 
0 0 2.5 
1 2 2.8 
3 4 3.0 
8 9 3.3 
15 16 3.5 
25 25 3.7 
39 39 3.9 
55 54 4.1 
71 70 43 
86 85 45 
95 95 4.7 
99 99 49 
99 99 5.0 
4.0 4.0 Avg. 
Doct. Mean 
0 0 1.5 
0 0 2.0 
0 0 2.5 
2 2 2.8 
4 4 3.0 
10 10 3.3 
17 18 3.5 
27 28 3.7 
42 43 3.9 
59 58 4.1 
716 74 43 
90 88 45 
97 96 4.7 
99 99 49 
99 99 5.0 
3.9 3.9 Avg. 
Doct. Mean 
0 0 1.5 
0 0 2.0 
0 0 2.5 
2 2 2.8 
4 5 3.0 
10 11 3.3 
17 18 3.5 
27 28 3.7 
40 40 3.9 
55 54 4.1 
71 69 43 
85 83 45 
94 93 4.7 
99 98 49 
99 99 5.0 
4.0 4.0 Avg. 


Table 9 is continued on the next page. 


22 


BA,BS MA,MS Doct. 


0 


0 


22. Principles, theories (adjusted) 


All 2-Yr_ BA,BS MA,MS_ Doct. 
0 0 0 0 
0 0 0 0 
1 1 2 1 
3 2 6 3 
5 4 9 5 
12 10 19 12 
20. «17 27 20 
32-28 39 31 
4744 53 46 
63 ~=«O6 69 62 
719 78 82 79 
90 90 91 90 
96 96 97 96 
99 99 99 99 
99 99 99 99 
39 40 3.8 3.9 
23. Applications (adjusted) 

All 2-Yr BA,BS MA,MS 
0 0 0 0 
0 0 0 0 
1 1 2 1 
3 2 B) 3 
6 4 8 6 
12 1 17 13 
20 »=«18 26 20 
31-29 36 30 
44 44 49 44 
596 64 59 
74 76 77 73 
86-88 88 85 
94 (95 95 94 
98 98 98 98 
99 99 99 99 
40 40 3.9 4.0 


24. Professional skills, attitudes (unadjusted) 


Percentile Ranks for IDEA Diagnostic Form Items and Scales 


Table 9 (continued) 


By Type of Institution 


All 2-Yr BA,BS MA,MS_ Doct. 


wWwNnNodode 


8 
14 
22 
33 
47 
63 
80 
91 
98 
99 

4.1 


skills (unadjusted) 


2-Yr BA,BS MA,MS 


0 


0 
0 


11 
18 
23 
31 
41 
55 
69 
81 
92 
98 
99 
3.9 


NNW Oe 


26. Creative capacities (unadjusted) 
2-Yr BA,BS MA,MS_ Doct. 


Mean 
1.5 0 
2.0 0 
2.5 0 
2.8 2 
3.0 4 
3.3 9 
3.5 15 
3.7 23 
3.9 35 
4.1 48 
43 64 
4.5 80 
4.7 91 
4.9 98 
5.0 99 
Avg. 40 
25. Team 
Mean All 
1.5 0 
2.0 0 
2.5 3 
2.8 6 
3.0 8 
3.3 15 
3.5 21 
3.7 30 
3.9 41 
4.1 54 
43 68 
4.5 81 
4.7 92 
4.9 98 
5.0 99 
Avg. 3.9 
Mean All 
1.5 0 
2.0 1 
2.5 4 
2.8 8 
3.0 12 
3.3 19 
3.5 26 
3.7 34 
3.9 45 
4.1 56 
43 68 
4.5 81 
4.7 91 
4.9 97 
5.0 99 
Avg. 3.9 


0 


0 


0 
2 
6 
10 
15 
23 
29 
37 
46 
56 
68 
80 
89 
97 
99 
3.8 
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24. Professional skills, attitudes (adjusted) 


Mean 


0 


All 2-Yr BA,BS MA,MS Doct. 
0 0 0 0 
0 0 0 0 
1 1 2 1 
3 3 4 3 
m) 5 7 5 

11 11 14 11 
18 = 19 21 18 
28 = 29 31 27 
41 44 43 39 
56 — (0 37 53 
71 75 71 68 
8487 84 82 
92 9A 93 92 
97 = «98 97 97 
98 99 98 98 

40 40 4.0 4.0 

25. Team skills (adjusted) 

All 2-Yr BA,BS MA,MS 
0 0 0 0 
1 1 1 1 
4 3 7 3 
7 9 11 6 
11 12 16 9 
19 20 24 17 

26 =—-28 30 24 
3538 38 33 
47 51 51 45 
61 = =64 65 58 
715 78 7] 72 
86 = 889 87 85 
93 95 94 92 
97 = «98 98 97 
98 _—*98 99 98 

3.9 38 3.8 3.9 

26. Creative capacities (adjusted) 

All 2-Yr BA,BS MA,MS_ Doct. 
0 0 0 0 
1 1 2 2 
5 3 6 7 
9 7 10 12 
13. ol 15 1S 

21 18 24 24 
29, 27 30 32 
3837 39 41 
48 49 49 50 
60 = 62 60 60 
722 #74 72 71 
83 85 82 81 
91 93 90 89 
96 = =97 96 95 
97 = «98 97 97 
3.9 39 3.8 3.8 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 
By Type of Institution 


27. Broad liberal education (unadjusted) 


Mean 
1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
4.7 
49 
5.0 

Avg. 


All 2-Yr BA,BS MA,MS Doct. 


0 0 0 0 0 
1 0 2 2, 2 
7 3 8 8 8 
13 7 15 15 15 
18 11 20 20 20 
28 = =20 30 30 30 
36029 40 38 38 
45 39 48 46 47 
5651 58 56 56 
65 62 67 66 65 
76 75 77 77 75 
8687 87 86 85 
94 95 95 94 93 
98 99 99 98 98 
99 99 99 99 99 
3.7 3.8 3.6 37 3.7 


28. Communication skills (unadjusted) 


Mean 
1.5 
2.0 
2.5 
2.8 


All 2-Yr BA,BS MA,MS_ Doct. 


0 0 0 0 0 
1 0 2 1 1 
4 =) 6 5 =) 
9 7 11 10 9 
13 11 16 14 13 
21 17 27 23 20 
29, 25 37 32 28 
39, 35 46 42 37 
50 47 56 54 47 
62 59 66 64 59 
715 73 76 77 71 
86 = 86 86 87 84 
94 95 94 95 93 
99 =99 98 99 98 
99 =99 99 99 99 
3.8 3.8 331 3.7 3.8 


29. Find, use resources (unadjusted) 


Mean 


1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
34 
3.9 
41 
43 
45 
47 
49 
5.0 
Avg. 


All 2-Yr BA,BS MA,MS Doct. 


0 0 0 0 0 

0 0 1 0 0 

2 1 6 2 2 

6 3 13 | 6 
10 7 19 12 11 
22 16 33 24 24 
32, 29 45 35 34 
44-38 Sf 48 45 
58 = 54 70 62 59 
71 68 81 74 71 
84-82 90 85 83 
92 92 95 93 91 
97 OT 98 97 97 
99 =99 99 99 99 
99 ~=99 99 99 99 
3.7 3.8 3.5 33] 3.7 
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27. Broad liberal education (adjusted) 


Mean All 2-Yr BA,BS MA,MS Doct. 


0 0 1 0 0 
2 1 5 3 3 
8 4 12 8 10 
14 8 18 15 17 
20 «14 25 20 23 
300-25 37 31 32 
39 34 44 39 41 
49 45 52 48 49 
59 57 61 58 59 
69 = 68 71 69 69 
79 ~=80 79 78 77 
87 = 88 87 86 86 
93 «94 93 92 93 
97 = (98 97 96 97 
98 98 98 97 98 
3.7 3.8 3.6 3.7 3.7 


28. Communication skills (adjusted) 


1.5 
2.0 
2.5 
2.8 


Mean All 2-Yr BA,BS MA,MS_ Doct. 


0 0 1 0 0 

1 1 3 2 2 
=) 4 9 5 6 
10 8 14 11 10 
14 #11 21 16 14 
24 20 33 27 23 
3328 42 36 32 
43 38 51 47 41 
54 50 61 57 52 
66 = 63 70 68 63 
77 75 78 79 74 
86 = 86 87 87 85 
93 «94 93 94 92 
97 = 97 97 97 97 
98 98 98 98 98 
3.8 3.9 33 33 3.8 


29. Find, use resources (adjusted) 
All 2-Yr BA,BS MA,MS Doct. 


0 0 0 0 0 
0 0 3 0 0 
3 1 10 3 3 
8 4 19 9 7 
12 8 25 14 13 
24 17 40 27 25 
35. 27 52 39 37 
47 40 64 52 48 
61 56 75 65 61 
74 71 85 77 72 
8585 91 86 84 
92 92 95 93 92 
97 = 97 97 97 96 
99 =99 99 98 99 
99 99 99 99 99 
37 3.8 3.4 3.7 3.7 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 


By Type of Institution 
30. Values development (unadjusted) 30. Values development (adjusted) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS Doct. 
LS 0 0 0 0 0 1.5 0 0 0 0 0 
2.0 0 0 1 0 0 2.0 1 0 2 1 1 
2.5 3 2 5 3 4 2.5 4 3 9 4 6 
2.8 7 a 10 7 9 2.8 9 6 15 9 11 
3.0 11 8 15 11 13 3.0 14 «10 21 13 16 
3.3 21 16 26 21 22 3.3 24 =19 32 24 26 
3.5 30.25 35 30 31 35 33-29 41 32 35 
3.7 40 37 45 40 41 ah 45 42 51 43 46 
3.9 53.51 56 51 53 3.9 5758 61 55 57 
4.1 65 65 67 63 63 4.1 70 = 72 69 67 69 
43 77 ~—-80 79 75 75 43 81 84 80 78 80 
45 88 90 89 87 87 45 89 =: 93 88 86 89 
47 95 96 96 94 94 47 95 97 94 93 95 
4.9 99 =—99 99 99 99 4.9 98 «498 97 97 98 
5.0 99 99 99 99 99 5.0 99 99 98 98 99 
Avg. 3.8 3.8 37 3.8 3.8 Avg. 38 38 3.7 3.8 3.7 
31. Critical analysis (unadjusted) 31. Critical analysis (adjusted) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS Doct. 
1.5 0 0 0 0 0 1.5 0 0 0 0 0 
2.0 0 0 1 0 0 2.0 0 0 2 0 0 
2.5 2 1 4 2 2 2.5 3 1 6 3 3 
2.8 5 3 9 6 6 2.8 7 4 12 7 8 
3.0 9 6 12 10 10 3.0 10 7 16 11 12 
3.3 17 13 20 18 19 3.3 20 «15 26 21 23 
3.5 25. 20 29 27 28 3.5 28 = 23 36 29 31 
3.7 35-30 40 37 37 3.7 40 35 47 41 41 
3.9 48 45 53 49 49 3.9 53.50 60 53 53 
4.1 62 59 65 63 61 4.1 67 ~—s 66 72 66 66 
43 76 = =75 78 76 74 43 80 =. 80 81 79 80 
45 88 =. 88 89 87 87 45 90 «(Ol 90 88 89 
47 95 95 96 95 95 4.7 95 96 95 94 96 
49 99 =99 99 99 99 4.9 98 = «98 98 98 98 
5.0 99 99 99 99 99 5.0 99 99 99 99 99 
Avg. 3.8 3.9 3.8 3.8 3.8 Avg. 38 3.9 3.7 3.8 3.8 
32. Interest in learning (unadjusted) 32. Interest in learning (adjusted) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS_ Doct. 
1.5 0 0 0 0 0 1.5 0 0 0 0 0 
2.0 0 0 0 0 0 2.0 0 0 1 0 0 
25 2 0 3 2 2 2.5 3 1 6 3 2 
2.8 5 3 7 = 6 2.8 6 4 12 7 yi 
3.0 9 =) 12 10 11 3.0 11 6 17 11 12 
3.3 18 11 23 20 22 3.3 21 14 31 23 24 
35 28 19 33 31 32 35 31 23 42 33 33 
3.7 40 30 4] 42 44 3.7 43, 35 54 45 45 
3.9 54.44 61 56 57 3.9 57.50 66 58 58 
4.1 67 ~— 60 74 70 69 4.1 71 66 78 72 70 
43 80 3975 85 82 80 43 82 ~=—- 80 88 83 82 
45 90 «88 94 91 90 45 91 8990 93 90 90 
47 96 896 98 97 96 4.7 96 86496 97 96 95 
4.9 99 =—99 99 99 99 4.9 98 = 498 99 98 98 
5.0 99 99 99 99 99 5.0 99 _—99 99 99 99 
Avg. 3.8 3.9 3.7 3.8 3.8 Avg. 38 3.9 3.6 3.8 3.8 
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Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 
By Type of Institution 


Progress on Relevant Objectives (unadjusted) 
(PRO ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10.) 


All 
0 


50.7 


2-Yr BA,BS MA,MS_ Doct. 


0 1 0 0 
2 3 2 2 
4 6 5 5 
10 13 12 12 
15 21 18 20 
21 27 24 25 
32 38 36 35 
41 47 44 43 
55 61 59 57 
66 71 69 67 
81 83 82 79 
88 89 89 87 
93 94 94 93 
98 98 98 98 
99 99 99 99 
51.3 50.0 50.5 50.8 


33. Amount of reading 


Mean All 2-Yr 


15 
2.0 
2D 
2.8 


Avg. 


2 

6 
16 
26 
35 
53 
65 
75 


1 

») 
16 
26 
34 
51 
64 
74 
82 


BA,BS MA,MS Doct. 


35. Difficulty 
Mean All 2-Yr 


0 
1 
=) 
12 
22 
43 
57 
69 
719 
86 
92 
96 
98 
99 
99 
34 


0 
0 
3 
9 
19 
40 
56 
69 
80 
87 
93 
96 
98 
99 
99 
3.5 


5 1 1 
14 6 5 
28 14 15 
38 23 25 
47 32 35 
60 51 53 
68 64 66 
76 74 75 
82 83 83 
88 89 89 
92 93 93 
95 96 97 
98 98 98 
99 99 99 
99 99 99 
3.0 3.2 3.2 

BA,BS MA,MS Doct. 

0 0 0 

2 0 1 

7 5 6 
16 13 14 
27 23 25 
47 43 46 
60 56 61 
72 68 73 
81 77 83 
88 85 89 
92 91 94 
96 95 97 
98 98 99 
99 99 99 
99 99 99 
33 34 34 
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Progress on Relevant Objectives (adjusted) 


Mean 


25 
30 
35 
40 
43 
45 
48 
50 
53 
=) 
58 
60 
62 
65 
70 


Avg. 


Mean 
1.5 
2.0 
2:5 
2.8 
3.0 
3.3 
3.5 
3.7 


All 2-Yr BA,BS MA,MS Doct. 
0 0 1 0 0 
2 1 3 1 2: 
4 4 6 4 5 

10 9 14 11 11 
17 15 22 17 18 
22 «21 28 23 23 
34 33 41 34 34 
3B 42 49 43 43 
58 58 63 58 57 
68 68 72 68 67 
81 82 84 81 80 
88 89 90 88 87 
93 94 94 93 92 
97 97 97 97 97 
99 99 99 99 99 
50.9 51.2 49.7 51.1 51.0 
34. Amount of other work 
All 2-Yr BA,BS MA,MS Doct. 
0 0 0 0 0 
0 0 2 1 1 
5 3 8 7 5 
13 8 16 17 13 
23 16 26 27 22 
41 34 44 46 42 
55 50 56 59 56 
68 65 67 71 68 
719 78 78 80 719 
86 87 86 87 87 
92 «=93 91 92 92 
96 8696 95 96 96 
98 99 97 98 98 
99 =99 99 99 99 
99 99 99 99 99 
34 3.5 3.4 3.4 3.4 


36. Strong desire to take course 


All 2-Yr BA,BS MA,MS Doct. 
0 0 0 0 0 

0 0 0 0 1 
4 3 5 5 5 
11 7 11 11 12 
1733 17 17 18 
29 = 25 28 30 31 
39 335 37 41 40 
50.) 45 48 52 51 
61 56 59 64 62 
71 66 69 74 72 
80 = 75 79 84 82 
898 88 91 90 
94 «91 94 96 96 
98 = 97 98 99 99 
99 —«99 99 99 99 
37 38 3.7 3.6 3.6 


37. Worked hard 
Mean All 2-Yr 


1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


Ooo 


BA,BS MA,MS Doct. 


39. Wanted course 


Mean All 2-Yr 


15 
2.0 
22 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


40. Increased positive attitude (unadjusted) 


0 


Mean 


1.5 
2.0 
2a 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 


0 


0 


BA,BS 


0 
0 
7 
18 
28 
48 
62 
75 
85 
92 
96 
98 
99 
99 
99 
33 


0 


MA,MS 


Doct. 


17 
28 
48 
63 
75 
86 
93 
97 
99 
99 
99 
99 
3.3 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items 
By Type of Institution 


All 2-Yr BA,BS MA,MS_ Doct. 


0 
0 
2 
5 
9 
17 
25 
35 
47 
60 
74 
85 
94 
98 
99 
3.9 


0 
0 
1 
4 
fj 
15 
23 
32 
45 
58 
72 
84 
93 
98 
99 
3.9 


BSERESRRRGCUNGO 


3.9 


0 


9 
18 
26 
37 
50 
63 
76 
88 
95 
99 
99 

3.8 
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ZT 


38. Wanted instructor 


Mean All 2-Yr 


1.5 
2.0 
2:5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


40. Increased positive attitude (adjusted) 


Mean 
1.5 
2.0 
2.5 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 


0 


0 
1 


All 2-Yr 
0 0 

0 0 

3 3 

7 7 
il 11 
21 21 
30.31 
41 43 
340 C*SS7 
67 ~—s 69 
78 ~=681 
87 ~—s 89 
93 (94 
97-97 
98 «2988 
3.8 3.8 


BA,BS MA,MS Doct. 


0 
1 
8 


0 
1 


0 

1 

9 
21 
31 
47 
58 
69 
77 
84 
90 
94 
97 
99 
99 
34 


BA,BS MA,MS_ Doct. 


0 
0 
3 


0 
1 


3 


= 
Woh re© 


41. Excellent teacher (unadjusted) 
2-Yr BA,BS MA,MS Doct. 


Mean 


1:5 
2.0 
2.5 
2.8 
3.0 
3:3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


A 


an 


1 


NRWRr CO 
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42. Excellent course (unadjusted) 


Mean 


1.5 
2.0 
23 
2.8 
3.0 
3.3 
3.5 
37 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


All 2-Yr BA,BS MA,MS 


0 


me) 
8 
15 
23 
32 
43 
56 
69 
82 
92 
98 
99 
3.9 


0 


3 
5 
11 
17 
25 
37 
50 
65 
80 
91 
98 
99 
4.0 


0 
0 
2 
b) 
8 
15 
23 
32 
43 
56 
69 
82 
92 
98 
99 
3.9 


43. Usually work hard 


Mean All 2-Yr 


1.5 
2.0 
2.5 
2.8 
3.0 
33. 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 
Avg. 


3.6 


3.6 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 
By Type of Institution 


0 0 
0 0 
2 2, 
4 5 
6 7 
11 12 
15 17 
21 23 
28 30 
37 40 
49 52 
63 64 
719 719 
93 94 
98 98 
4.2 4.1 
Doct. 

0 0 
0 0 
2, 3 
5 6 
8 10 
17 19 
24 27 
34 36 
46 47 
59 59 
72 72 
84 83 
93 92 
98 98 
99 99 
3.9 3.9 


BA,BS MA,MS Doct. 


3.7 


3.6 


Table 9 is continued on the next page. 
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1:5 
2.0 
29 
2.8 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
4.9 
5.0 
Avg. 


41. Excellent teacher (adjusted) 


Mean All 2-Yr BA,BS MA,MS Doct. 


0 0 0 0 0 
0 0 1 0 0 
2 1 ) 2 3 
4 3 6 4 6 
6 5 8 7 8 
11 9 13 12 14 
16 «13 17 17 19 
22 «18 24 22 26 
29, 25 31 30 34 
40 35 41 40 44 
52. 48 53 52 57 
67 = 64 67 66 70 
81 = 80 82 80 84 
929d 93 91 94 
96 95 96 95 96 
42 42 4.1 42 41 


42. Excellent course (adjusted) 
All 2-Yr BA,BS MA,MS Doct. 


0 0 0 0 0 

0 0 0 0 1 

3 2 3 3 4 

6 4 7 6 8 
10 8 10 10 13 
18 «15 19 19 23 
26 23 27 27 31 
3634 37 36 41 
48 47 49 48 53 
61 60 61 60 65 
74 73 73 73 76 
84 684 84 83 86 
929d 92 91 92 
96 96 97 96 96 
97 = OT 98 97 98 


44. Variety teaching methods 


Mean All 2-Yr BA,BS MA,MS Doct. 


15 
2.0 
2.5 
28 
3.0 
3.3 
3.5 
3.7 
3.9 
4.1 
43 
45 
47 
49 
5.0 


Avg. 


0 0 0 0 0 
0 0 1 1 1 
3 2 2 3 3 
6 4 5) 7 6 
9 7 8 10 10 
17 14 16 18 18 
24 22 23 26 26 
3532 34 37 37 
48 47 47 50 51 
63 «62 62 65 65 
77 78 7] 79 78 
89 = 90 89 90 89 
96 8696 96 97 96 
99 «99 99 99 99 
99 «99 99 99 99 
3.8 3.9 3.8 3.8 3.8 


Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 


By Type of Institution 
45. Students given responsibility 46. High achievement standards 

Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS Doct. 
15 0 0 0 0 0 [5 0 0 0 0 0 
2.0 0 0 0 0 0 2.0 0 0 0 0 0 
2.5 0 0 0 0 0 25 0 0 0 0 0 
2.8 0 0 0 0 0 2.8 0 0 0 0 0 
3.0 0 0 0 0 0 3.0 1 1 1 0 1 
3.3 0 0 0 0 0 3.3 3 2 4 2 4 
3.5 1 1 1 1 2 3:5 7 6 9 6 9 
3.7 4 4 4 3 5 3.7 14 12 16 13 18 
3.9 II 11 11 10 14 3.9 27 25 29 27 32 
41 25 24 25 24 28 41 44 42 47 44 49 
43 46 45 47 47 50 43 64 63 66 64 67 
45 71 71 72 72 73 45 81 81 81 81 83 
47 89 89 90 90 90 47 92 93 92 92 93 
49 98 98 98 98 98 49 98 98 98 98 99 
5.0 99 99 99 99 99 5.0 99 99 99 99 99 
Avg. 43 43 43 43 43 Avg. 41 41 4.1 41 41 

47. Used educational technology Stimulating Student Interest (4 items) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS._ Doct. 
15 0 0 0 0 0 10.0 0 0 1 0 1 
2.0 2 1 4 3 2 11.0 2 1 2 2 2 
2.5 9 6 13 11 9 12.0 4 3 °) 4 5 
28 16 12 22 18 15 13.0 8 7 9 8 11 
3.0 21 18 29 25 21 14.0 16 = 13 17 16 19 
3.3 32 28 41 36 31 15.0 28 = 24 30 29 31 
35 40 37 49 44 39 15.5 36-32 38 38 39 
3.7 49 46 57 53 48 16.0 45 40 48 47 48 
3.9 58 56 66 62 58 16.5 5551 59 58 57 
41 68 66 74 70 67 17.0 6 61 69 68 66 
43 77 76 81 78 77 17.5 75 72 79 78 75 
45 86 86 89 86 86 18.0 8482 87 86 83 
47 93 93 95 93 93 18.5 91 90 93 92 90 
49 998 98 98 98 98 19.0 96 95 97 97 95 
5.0 99 99 99 99 99 20.0 99 99 99 99 99 
Avg. 36 37 33 3.6 36 Avg. 15.9 162 15.8 15.9 15.8 

Fostering Student Collaboration (3 items) Establishing Rapport (4 items) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS_ Doct. 
5.0 0 0 0 0 0 10.0 0 0 1 0 1 
6.0 1 1 1 2 2 11.0 1 1 2 1 2 
7.0 5 4 5 ) 5 12.0 3 3 3 3 4 
8.0 li 10 11 12 11 13.0 7 6 7 7 9 
9.0 19 19 20 22 18 14.0 14 12 14 14 16 
10.0 30 =: 30 32 33 28 15.0 25 23 25 25 27 
11.0 44 44 47 47 41 15.5 32-330 34 33 35 
11.5 52. 53 56 55 48 16.0 41 40 43 42 43 
12.0 61 62 66 63 58 16.5 S150 54 52 53 
12.5 70 72 76 72 67 17.0 62 62 65 63 63 
13.0 79 8i 84 81 76 17.5 73 8673 76 74 72 
13.5 88 = 89 92 89 85 18.0 83-83 86 84 82 
14.0 94 = 95 96 95 93 18.5 91 Ot 92 92 90 
14.5 98  =—-98 99 98 97 19.0 96 96 97 97 95 
15.0 99 «99 99 99 99 20.0 99 99 99 99 99 
Avg. 110 11.0 10.8 10.9 111 Avg. 161 162 16.0 16.1 16.0 


Table 9 is continued on the next page. 
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Table 9 (continued) 
Percentile Ranks for IDEA Diagnostic Form Items and Scales 


By Type of Institution 
Encouraging Student Involvement (4 items) Structuring Classroom Experience (5 items) 
Mean All 2-Yr BA,BS MA,MS Doct. Mean All 2-Yr BA,BS MA,MS Doct. 
10.0 1 1 1 1 1 13.0 0 0 0 0 1 
11.0 3 3 4 4 4 15.0 2 1 2 2 3 
12.0 7 6 8 8 8 17.0 6 e) 7 7 8 
13.0 130 «12 13 14 14 18.0 11 8 12 11 14 
14.0 22. 20 23 23 23 19.0 18 = 14 19 19 23 
15.0 3432 36 36 35 20.0 28 = 23 30 30 34 
15.5 41 40 45 44 41 20.5 35-29 38 37 41 
16.0 49 48 53 53 48 21.0 43 37 46 45 49 
16.5 58 58 62 61 56 21.5 52. 45 56 54 58 
17.0 67 — 68 71 70 65 22.0 61 55 66 64 66 
17.5 76 477 80 78 73 22.5 71 = «66 77 75 75 
18.0 84 = 886 88 86 81 23.0 81 77 86 84 83 
18.5 91 92 94 92 89 23:5 89 87 93 91 90 
19.0 96 96 97 96 95 24.0 95 94 97 96 95 
20.0 99 99 99 99 99 25.0 99 99 99 99 99 
Avg. 15.6 15.7 15.5 15.5 15.7 Avg. 209 21.3 20.7 20.8 20.6 


Average ratings were generally about the same for institutions of various sizes (less than 1000; 1000- 
2499; 2500-4999; 5000-9999; and 10,000+). Of the 47 items, differences in average ratings among 
these groups exceeded 0.1 on only 12. Results for these 12 items are shown in Table 10. 


Table 10 
Average Ratings by Institutional Size on TweWe Items 
Institutional Size 


5. Formed “teams” or “discussion 


groups” 
11. Related course to real life situations 


25. Progress on “team skills” 

26. Progress on “creative capacities” 

29. Progress on “finding, using 
resources” 

33. Amount of required reading 


On most of these items, average ratings for institutions with the smallest enrollments tended to be 
lower than those for larger institutions. However, on an overall basis, the differences were too slight 
to conclude that institutional size had a significant influence on ratings. 
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II. The Structure of the Ratings 


Although students and faculty both rate 12 learning objectives, it is possible that a smaller 
number of “dimensions” would be adequate to describe “goals” or “progress.” Similarly, 
student ratings of 20 teaching methods may well represent fewer than 20 teaching “styles.” 


To determine if there was a meaningful underlying structure to either the ratings of 
objectives or ratings of teaching methods, three Maximum Likelihood Factor Analyses with 
Orthogonal Rotation* were conducted. One of these was for faculty ratings of the 
importance of the 12 objectives; a second was for student ratings of progress of these 
objectives; and the third was for student ratings of teaching methods. Results for both the 
Short and Diagnostic Forms were used in these analyses. 


In all analyses, factors with eigenvalues greater than 1.0 were extracted and rotated by the 
Varimax method. Rotated factor loadings of faculty ratings of the importance of the 12 
objectives are shown in Table 11. 

Table 11 


Rotated Factor Loadings for 
Faculty Ratings of the Importance of Objectives 


11. Learning to analyze and critically evaluate ideas, 
arguments, and points of view 

12. Acquiring an interest in learning more by asking questions 
and seeking answers 


questions or solving problems ; j ; 
personal values , ; ; 

7. Gaining a broader understanding and appreciation of famed 
intellectual/cultural activity (music, science, literature, etc.) 

6. Developing creative capacities (writing, inventing, designing, 
performing i in art, music, drama, etc.) 


eee [ole le 

o psrofessionnls 3 in the fields related to this course 

5. Acquiring skills in working with others as a member of a 

3. Learning to apply course material (to improve thinking, 
problem solving, and decisions) 

1. Gaining factual knowledge (terminology, trends, etc) P10 06.1) = 6 


Although the structure that emerged from this analysis was somewhat ambiguous, there 
were three relatively clear groupings of objectives. The first loading principally on Factor I, 
and included (in abbreviated form) Critical analysis, Interest in learning, Values 


* Lawley, D. N. (1940) “The Estimation of Factor Loadings for the Method of Maximum Likelihood,” 
Proceedings/The Royal Society of Edinburgh, 60, 64-82. Kaiser, H. F. (1958), “The Varimax Criterion for 
Analytic Rotation in Factor Analysis,” Psychometrika, 23, 187-200. 

31 


development, Broad liberal education, and Communication skills. Taken together, these 
objectives seem to emphasize /ntellectual Development. 


Three other objectives loaded primarily on Factor I—Professional skills, viewpoints; 
Applications; and Team skills. The common focus of these objectives appears to be 
Professional Preparation. 


Finally, two objectives loaded primarily on Factor [17—Principles and theories and Factual 
knowledge. These objectives both stress Basic Cognitive Development. 


The other two objectives (Creative capacities; Finding and using resources) appeared to 
represent a combination of Factor I (intellectual Development) and Factor II (Professional 
Skills). Conceptually, then, faculty objectives centered on Basic Cognitive Development, a 
broader Intellectual Development, or Professional Preparation; but two objectives appeared 
to combine the last two of these. 


Did student ratings of their progress parallel faculty ratings of importance? Table 12 
explores this question. 


Table 12 
Rotated Factor Loadings for 
Student Ratings of Progress on Objectives 


ee Factor | Factor 


8. Developing skill in expression myself orally or in writing 
6. Developing creative capacities 


11. Learning to analyze and critically evaluate ideas, arguments, and 
points of view TD 45 


* Seleuaealulaetvty Gene, Seacyete) | 78 | a8 
intellectual/cultural activity (music, science, etc.) 73 26 


4, Developing professional competencies, points of view 
12. Acquiring an interest in learning more | 63 | 66 | 


In this analysis, only two factors were extracted. The structure of progress ratings appears 
generally different from that of faculty “importance” ratings. The one clear similarity 
between the two involves the two objectives that had high loadings on Factor II but low 
ratings on Factor I in Table 12 (Principles and theories; Factual knowledge). This was 
called Basic Cognitive Development in the previous analysis, and might be labeled Building 
a Cognitive Background in the present analysis. 


All other objectives had substantial loadings on Factor I, ranging from .43 to .91, together 
with a wide range of loadings on Factor II. It can be inferred that all were perceived to 
involve cognitive development in addition to some other kind of development, represented 
by the Factor II rotated loading. An examination of the rotated loadings on both factors 
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suggests that various combinations of these loadings represent different ways students use 
their backgrounds to advance educational competencies: 


1. Professional Development (Objectives 3 and 4; loadings on Factors I and II of 
.44/.79 and .43/.78, respectively). 

2. Intellectual Development (Objectives 7, 10, and 11; loadings on Factors I and II 
were .73/.26, .75/.44, and .75/.45, respectively). 

3. Expressiveness (Objectives 6 and 8; loadings of .85/.19 and .91/.17). 

4. Life Long Learning Skills (Objectives 5, 9, and 12; loadings of .59/.30, .62/.53, 
and .63/.66). 


Although the terminology suggested by the analysis of student ratings is similar to that used 
in describing faculty ratings, the two analyses do not always agree on the placement of 
individual objectives. They did agree that Basic Cognitive Development is being stressed by 
the first two objectives and that the third and fourth objectives related to Professional 
Development. Furthermore, Objectives 7, 10, and 11 were classified as Intellectual 
Development in both analyses. But Expressiveness and Life-Long Learning Skills, which 
seemed to emerge from the student analysis, were not evident as separate dimensions in the 
faculty ratings. 


It can be concluded that conceptualizations of faculty aspirations and student perceived 
outcomes have much in common. Both agree that conceptualization should include Basic 
Cognitive Development, Professional Development, and Intellectual Development. Student 
ratings offer two additional ways of conceptualizing the advancement of educational 
competencies—Expressiveness and Life Long Learning Skills. It should be noted that the 
two objectives not readily classified in the faculty analysis were included in the last two 
dimensions of the student analysis (Creative capacities as an Expressiveness objective and 
Finding, using resources as a Life Long Learning objective). 


It appears that the first two objectives are sufficiently redundant that, in subsequent revisions 
of the instrument, they could be combined. Other than that, the mathematical structures that 
emerged from these analyses were not very crisp. They may provide some guidance to 
those interested in developing conceptual schemes for describing the purposes of higher 
education, and will be used to classify the objectives in the IDEA Center’s Directions to 
Faculty. But they provided no reason to alter the current focus of the IDEA system on the 
relative importance of each individual objective. 


The final factor analysis was performed on student ratings of the 20 instructional methods 
Two factors were extracted. Rotated factor loadings are shown in Table 13. 


ele) 


Table 13 
Rotated Factor Loadings for Student Ratings of Instructional Methods 
Method Factor II 
25 


Go] Go 
Bin 


— 
Nn 


12. Gave tests etc. that covered most important points 


5. Formed “teams” or “discussion groups” 
14. Involved students in “hands on” experiences 27 75 


<a) 
n 


An examination of the rotated factor loadings suggests that the first factor focuses on the 
instructor’s role in transmitting knowledge while the second emphasizes the student’s role in 
acquiring knowledge. 


Within these broad categories, subgroups of items can be formed by attending to the relative 
size of the rotated loading on the two factors. The first subgroup (high loadings on Factor I; 
relatively low loadings on Factor II) appears to emphasize providing a clear classroom 
structure; the focus seems to be on course content. The next two item subgroups appear to 
center on increasing student motivation, a potent influence on learning. One aspect of 
motivation is reflected in the second subgroup (relatively high loadings on Factor I; 
moderate loadings on Factor II), which features ways of stimulating student interest. The 
four items in the next subgroup (where loadings on the two factors were nearly equal) 
emphasized a related approach to improving student motivation—methods designed to 
stimulate student effort. Although attracting interest in the subject is often the first step in 
motivating students, additional efforts may be required to encourage the student effort that 
learning requires. 


The final two subgroups both have high loadings on Factor I, the factor stressing the 


student’s role in learning. The first stresses involving students in learning activities, it 
reflects the adage that the best way to learn something is to teach it. The second emphasizes 
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student interaction; activities requiring the exchange of student views or team participation 
represent another way instructors may facilitate learning. 


Although the high inter-correlations among methods items resulted in a somewhat 
ambiguous factor structure, the sub- groupings of items make intuitive sense. Effective 
instruction requires attention to content; faculty members need to be not only authorities in 
their field but expert in organizing and communicating that content. Especially in lower 
division undergraduate courses, where student motivation is often low or marginal, the 
effective instructor must also attend to student readiness to learn, both by finding ways to 
capture student interest and by stimulating student effort. Although at times teaching is 
necessarily centered on the instructor’s input, effective instructors know that student 
learning is as much a function of what the student does as how the instructor proceeds. 


These “dimensions of effective teaching” are clearly not independent; a fact reflected in both 
the high item inter-correlations and the somewhat ambiguous factor structure. Classroom 
observations are consistent with this conclusion. Effective teachers typically organize and 
present class content. But at the same time, and sometimes with the same techniques, they 
elicit student interest, encourage student effort, and involve students in the teaching- learning 
process. It may be unwise and fruitless to conceptualize the “art” of teaching as a series of 
discrete and unrelated techniques. 


Prior to the conduct of these analyses, IDEA staff had proposed five a priori scales be 
developed using the 20 standard methods items. These scales were modeled after those 
developed by The National Survey of Student Engagement (NSSE)° to describe features of 
the campus environment which promote student learning. Because the IDEA scales were 
limited to the classroom environment, and because they had not been empirically developed, 
they were given slightly different names than those employed by NSSE. They were called 
Stimulating Student Interest, Fostering Student Collaboration, Establishing Rapport, 
Encouraging Student Involvement, and Structuring the Classroom. The similarity of these 
names to those suggested for the five subgroups produced by the factor analysis is obvious, 
even though there was only a moderate overlap among the specific items included on 
“scales” with similar names. Although there would be a modest statistical advantage in 
revising the content of these scales in accordance with findings from the factor study, the 
advantages gained by refining the scales was judged to be outweighed by the disadvantage 
of sacrificing longitudinal comparisons. 


In summary, results from the factor analyses were relatively ambiguous. When methods 
were analyzed, five alternative approaches to instruction were identified. These approaches 
were far from independent, suggesting that the effective instructor must be prepared to 
adjust strategies to different times and circumstances. The analyses of objectives show that, 
while they could be grouped into a smaller number of categories, these groupings were not 
entirely distinct. Therefore, it seems advisable (with the possible exception of objectives 
concerned with basic cognitive development) to continue having instructors select the 
pattern of objectives that best describes their intentions without regard for how these 
objectives relate to each other. 


> National Survey of Student Engagement. National Benchmarks of Effective Educational Practice. Indiana 
University Center for Postsecondary Research and Planning: Bloomington, Indiana, 2001. 
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III. The Process of Adjusting Ratings 


Teaching effectiveness is assessed in three ways—(1) the ratings of progress on individual 
objectives chosen as important or essential by the instructor; (2) the weighted average for 
objectives chosen by the instructor (Progress on Relevant Objectives - PRO); and (3) the 
three global measures (averages on As a result of taking this course, I have more positive 
feelings toward this field of study; Overall, I rate this instructor as an excellent teacher; and 
Overall, I rate this an excellent course. Effectiveness is reported in two ways—the simple 
average of student ratings on the measure and an “adjusted” measure. This section describes 
how “‘adjusted” scores were developed. 


Ratings are adjusted to take into account, insofar as possible, the fact that matters influence 
them that are beyond the instructor’s control. For example, if the majority of students were 
strongly motivated to take a class, ratings are likely to be higher than in classes with less 
interested students. Therefore, unless this is taken into account, instructors of highly 
motivated students would have an unfair advantage over those whose students were less 
interested and dedicated. 


In addition to size of class, the Diagnostic Form contains a number of items that are 
potentially relevant as measures of “extraneous circumstances.” The most apparent ones are 
Items 39 and 43 (/ really wanted to take this course regardless of who taught it; As a rule, I 
put forth more effort than other students on academic work.) For convenience, scores are 
these items are called “Course Motivation” (CM) and “Work Habits” (WH), respectively. 


Three other items were considered as relevant to potentially important extraneous 
circumstances—average ratings of Items 35, 36, and 37 (Difficulty of subject matter; I had a 
strong desire to take this course; and I worked harder on this course than on most courses I 
have taken). However, scores on these items could not be used as direct measures of 
extraneous influences because, at least in theory, each of them was, to a degree, under the 
control of the instructor. Obviously, the instructor controls many factors that make a course 
difficult or easy. Similarly, instructors can influence the amount of effort a student puts into 
a course. And, at least for some students, the desire to take a course may reflect the 
reputation its instructor has earned, a factor under the instructor’s control. 


Although ratings on these three items can be traced, in part, to instructor behavior or 
characteristics, they may also reflect factors that are not under the instructor’s control. 
Course difficulty may, for example, reflect the fact that disciplines differ on the degree to 
which they stress content that is inherently difficult (complex, obscure). Similarly, students 
may have a strong desire to take a course for reasons unrelated to the instructor’s reputation 
or behavior (the time of day the course was offered, the intent of friends to take the course, 
the need to satisfy some pre-requisite, etc.). And student effort may reflect, in addition to 
factors under the control of the instructor, such extraneous motivations as desire to be 
accepted in a professional school; desire to earn academic honors (or avoid academic 
dismissal); desire to impress someone else; etc. 


To determine whether ratings on any of these items represented extraneous influences that 
ought to be included in the adjustment process, an effort was made to exclude the portion of 
variation that could be accounted for by instructor behavior. The procedure was to conduct 
step-wise multiple regression analyses° that employed each of these three measures as the 
dependent variable. For two of the items (difficulty and effort), 22 independent variables 


Hocking, R. R. (1976) “The Analysis and Selection of Variables in Linear Regression,” Biometrics, 32, 1-50. 
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were employed (the 20 teaching methods items plus Items 33 and 34—Amount of reading 
and Amount of other work. For Item 36 (I had a strong desire to take this course), Item 38 
(I really wanted to take a course from this instructor) was used as the independent variable. 
This permitted us to predict average ratings on each of these three items on the basis of 
averages for the independent variables. 


This prediction represented the average rating expected on the basis of relevant student 
characteristics. By subtracting the prediction from the obtained average, we obtained a 
residual that represented the average on the item after the instructor’s influence had been 
removed. These residuals were labeled Dy (difficulty unrelated to the instructor), Ey (effort 
unrelated to the instructor), and OM (other motivation). A positive residual means that the 
average rating was higher than would be expected on the basis of the independent 
variable(s). In other words, after the influence of the instructor’s approach to the class had 
been taken into account, student ratings of effort and difficulty were above average. The 
“difficulty” residual probably reflects differences among disciplines; some are inherently 
more challenging than others to the majority of students. The “effort” residual may reflect 
the adequacy of student background and/or student academic self-confidence. 


In initial analyses, 7 independent variables made significant contributions to the prediction 
of Item 35 (difficulty); the same was true for Item 37 (Effort), although only 5 of the 7 
significant variables were identical. In both instances, the partial regression weight for two 
of the measures was negative, a finding that invariably obscures interpretation. 

Furthermore, the amount of variance accounted for by two other measures was less than two 
percent of the total. 


In the interest of simplicity, new analyses were undertaken which employed only the three 
most important measures. For both difficulty and effort, these were the average ratings on 
Items 33 (amount of reading), 34 (amount of other work), and 8 (stimulating intellectual 
effort). The formula for predicting “difficulty” was: 


Predicted X35 = .13412 Xz + .23986 X33 + .40303 Xx + .74331; R’ = 371 
Dn = Mean of X35 — Predicted X35 


For “effort,” these formulas were: 


Predicted X37 = .35690 Xz + .11142 X33 + .51595 X34+.06562; R? = .635 
En = Mean of X37 — Predicted X37 


Both formulas are easy to understand; the more reading is required, the more “other work” is 
required, and the more the instructor is perceived to stimulate intellectual effort, the more 
difficult the course is perceived to be and the more effort students report putting forth. Dy 
and Ey tell us whether the difficulty and effort reported by students was more (positive 
residual) or less (negative residual) than was expected on the basis of instructor-controlled 
factors. 


Other motivation (OM) was calculated by predicting the mean for Item 36 (J had a strong 
desire to take this course) from the mean of Item 38 (/ really wanted to take a course from 
this instructor) and subtracting the result from the obtained mean on Item 36. The formula 
was: 


Predicted X36 = .57366 X39 + 1.71732; R? = .327 
OM = Mean of X36 — Predicted X36 
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These results indicate that the desire to take a course can be partially explained by the desire 
to be exposed to a particular instructor. But a substantial portion of the variability in this 
measure is apparently due to other (unspecified) motivations. 


The next step in the adjustment process was to conduct step-wise multiple regression 
analyses which employed the 12 ratings of progress and the 3 global ratings as dependent 
variables and six independent variables—enrollment (N), CM (mean of Item 39), WH (mean 
of Item 43), Dx, En, and OM. When this was done, the OM measure was statistically 
significant in only two analyses; and in these two, it contributed less than 1 percent to the 
explained variance. Therefore, this measure was dropped and analyses were repeated using 
only five independent variables. 


Table 14 provides information about statistically significant regression weights and other 
data needed to compute adjusted scores. Appendix B shows calculations for an example. 


Table 14 
Regression Coefficients and Constants for Adjusting Ratings On the Diagnostic Form 


Regression Coefficient 
stant 


| 21. Factual knowledge | 1.69981 | 27568 | 38141 | | 09434 | -.07217 | 1.176 | 4.0013 | 


22. relied and 1.67498 | .25225| 39835 | -.00065 | .09683 | -.12443 | 1.163 | 3.9443 


ot Eton sa, 1.45513 | 32015 | .42804 | -.00284 | -.09290 | -.06913 | 1.238 | 4.0420 
viewpoints 


| 25.Team skills ___|_1.36271 | 20224 | 51612 |_| -.26412 | -.11336 | 1.161 | 3.9285 | 


27. Broad liberal 1.12469 | .24898 | .51462 | -.00463 | -.28984 | -.14497| 1.165 | 3.6948 
education 
28 ae 2.17413 | .03283 | .44629 | -.00774 | -.57321 po 1.193 | 3.7887 


-.19646 
| 30. Values development _| 1.15089 | 25370 | 47874 |_| -.24761 
-.10333 


pak ee ce 1.00177 | .51242 | .33205 | -.00113 | -.22342 | .07431 | 1.361 | 3.8611 
attitude 


41. Excellent teacher -.14747 
1.35036 -.00136 | -.21410 | .05304 3.9198 


CM=Course Motivation (item 39), WH=Work Habits (item 43), N=enrollment, Dy=Difficulty unrelated to the 
instructor, Ey=Effort unrelated to the instructor 
Note: Analyses reported in Table 14 are based on a more restricted data set. Classes with response rates less 
than 75% or not reporting the number enrolled were also excluded. 


It is clear from this table that “Work Habits” (WH, mean of Item 43) was generally the most 
potent predictor, followed by “Course Motivation” (CM, mean of Item 39). Classes that 
contained students who typically worked hard on their studies and/or were highly motivated 
to take the course regardless of who taught it were expected to receive favorable ratings; 
unless ratings were adjusted, the instructors of such classes would have an unfair advantage 
over colleagues with less motivated and less dedicated students. 


The joint effect of these two variables is displayed In Table 15. Classes were sorted into 5 
groups on the basis of average scores on Item 39 (course motivation). The “Low” group’s 
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average was in the lowest 10 percent of all averages; “Low Average” was in the next 20 
percent; “Average” was in the middle 40 percent; “High Average” in the next 20 percent; 
and “High” in the upper 10 percent. Then each of these groups was sorted into five 
similarly defined groups on the basis of their average response to Item 43 (work habits). 
The resulting 5x5 matrix produced 25 groups. Average progress ratings on each of the 12 
learning objectives for these 25 groups are shown in the table. The only classes included in 
this table were those for which the instructor identified the objective as “important” or 
“essential.” 


As seen in Table 15, the influence of these two variables on progress ratings is dramatized 
by comparing the two extreme groups (“Low/Low” vs. “High/High”). Differences ranged 
from 0.62 (for Communication Skills) to 1.17 (for Professional skills and viewpoints), 
averaging 0.96. Clearly, instructors in “High/High” classes have an enormous advantage 
over those in “Low/Low” classes; adjusted scores attempt to compensate for this advantage. 


39 


Table 15 
Average Progress Ratings for Classes That Differ in Levels of 
Student Motivation (Item 39) and Student Work Habits (Item 43) 


21. Gaining factual knowledge 27. Broad liberal education 


Work Habits Student Motivation (Item 39) Work Student Motivation (Item 39) 


(Item 43) Low High Habits Low High 
Low | Avg. | Avg. | Avg. | High (Item 43) | Low | Avg. | Avg.| Avg. | High 
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Low Avg 


22. Principles, theories 28. Communication skills 


Work Habits Student Motivation (Item 39) Work Habits Student Motivation (Item 39) 


(Item 43) High (Item 43) Low High 
Avg.| Avg. | Avg. | High Low | Avg.| Avg.| Avg. | High 
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58 | 3. 
69 | 3. 
95 | 4.10 


23. Applications 29. Finding and using resources 


Work Habits Student Motivation (Item 39) Work Habits Student Motivation (Item 39) 


(Item 43) Low High (Item 43) Low High 
Low | Avg.}| Avg. | Avg. | High Low | Avg.| Avg.| Avg. | High 


30. Values development 
Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
Low | Avg.| Avg.| Avg. | High 


31. Critical analysis 
Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
Low | Avg.| Avg.| Avg. | High 


32. Interest in continued learning 
Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
Low | Avg.| Avg.| Avg. | High 
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Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
ow | Avg.| Avg.| Avg. | High 


Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
Low | Avg.}| Avg. | Avg. | High 


A9 | 3.58 


Work Habits Student Motivation (Item 39) 


(Item 43) Low High 
Low | Avg.| Avg. | Avg. | High 


4.31 | 4.03 


The regression coefficient for “Enrollment” (N) was not always statistically significant; but 
when it was, it was always negative, meaning the larger the class, the lower the predicted 
(expected) rating. Those teaching small classes have an advantage over those teaching large 
classes; hence, in the interest of fairness, ratings should be adjusted to take this into account. 


Except for the first two criterion ratings, the regression coefficient for Dx was always 
negative. Generally, if the discipline was perceived as difficult (after taking into account the 
impact of the instructor on perceived difficulty), an attenuated outcome can be expected. 
This was especially apparent in progress ratings on “Creative capacities” and 
“Communication skills” where high difficulty was strongly associated with low progress 
ratings. The two exceptions, where “disciplinary difficulty” had a positive effect on the 
predicted outcome, were for the progress ratings concerned with basic cognitive 
development (“Factual knowledge” and “Principles and theories”). Consistent with other 
research regarding the influences of difficulty, this finding refutes conventional wisdom 
(high difficulty=low ratings). 


In most cases, student effort in the class (adjusted for the instructor’s influence on effort) 
was also negatively related to predicted ratings. Classes containing an unusually large 
number of students who worked harder than the instructor’s approach required ended up 
with lower progress ratings. As noted earlier, this may be because those who found it 
necessary to put in extra effort were those whose backgrounds did not prepare them well for 
the class. They may also be students who lack self-confidence and, for this reason, under- 
achieve (or under-estimate their progress in a self-abasing manner). 


Adjustments for the three global ratings merit special scrutiny. Regression results for 
predicted scores on “Increased positive attitude” and “Excellent course” were similar to each 
other. The order of the most influential predictors was reversed over that found for 
individual progress ratings; CM (desire to take the course regardless of who was teaching it) 
was the clear leader, and WH (tendency to work hard in academic studies) was a relatively 
distant second. Classes perceived as very difficult (DN) were generally rated low on these 
measures, but (again in contrast to the findings for individual progress ratings) those with 
substantial numbers of students who worked hard in the class generally rated it more 
favorably. In other words, when students worked harder than required by the instructor, 
they tended to have good impressions of both the discipline and the course, even though 
their ratings of progress on relevant objectives tended to be low. But both global ratings and 
specific progress ratings tended to be low in disciplines perceived to be inherently difficult. 


The other global rating (“Excellent instructor”) was not predicted with much accuracy 
(R*=.0883); these measures of extraneous influences were not very predictive of students’ 
overall impressions of their instructors’. Although significant regression weights were 
found for all five independent variables, these were all of modest magnitude. CM and WH 
were about equal in their influence on such ratings, while the adjusted ratings for 
“Difficulty” and “Effort” had a more moderate (and negative) influence. Enrollment size 
had a very minor and negative influence. Thus, instructor “popularity” was not accurately 
predicted by these measures; but student motivation and dedication did have a moderate 


’ Conceivably, this may be because ratings of this characteristic are determined almost exclusively by 
instructor behavior rather than by extraneous circumstances. Ratings on Item 10 Explained course material 
clearly and concisely, correlated .90 with overall ratings of the instructor (Item 41). See Table 6. 
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positive influence while disciplinary difficulty and student effort had a slight negative 
influence. 


The formula for adjusting means for progress ratings (Items 21-32) and global ratings (Items 
40-42) is Grand Mean + (Obtained Mean — Predicted Mean)*(1 + R’). This formula 
produces adjusted values with approximately the same mean and standard deviations as 
those obtained for unadjusted measures. 


Adjustments to ratings on the Short Form were less precise because it provided no 
information on WH, Dy or Ey. Since WH (work habits) was the most potent measure of 
relevant extraneous circumstances, its omission from the Short Form was especially 
regrettable. In later versions of this instrument, this item will be added. Until that time, it 


was decided to retain the adjustment formulas and process that have been in place since the 
1998-99 school year. 


The formula for predicting OM (other motivation) was developed from Short Form results; 
it is similar to, but not identical with, that reported earlier for the Diagnostic Form. 


Predicted Mean of Item 13 = .519087 X,4 + 1.804711 
OM = Mean Item 13 — Predicted Mean, Item 13 


Table 16 provides information regarding regression coefficients and constants used in 
adjusting Short Form scores. 
Table 16 
Regression Coefficients and Constants for Adjusting Ratings On the Short Form 
Grand 

|1.Factualknowledge | 2.83473 _| .32094 | -.06596| ---_| 1.102 | 3.9038 | 
| 2. Principles and theories | 3.07102_| .23693[_---_ | --_| 1.084 | 3.8526 | 
| 4. Professional skills, viewpoints | 3.00560_| .30163 | ---__| -.00262 | 1.117 | 3.9764 | 
| 6. Creative capacities | 3.18263 | 23181 | ---__| -.00504 | 1.070 | 3.8348 | 
"7. Broad liberal education | 3.12332_[.19650| | -.00326 | 1.034 [ 3.6707 
|12.Interestin learning | 3.15930_| .16133[ -.15513] -—-_| 1.011 | 3.7065 | 
| 16. Increased positive attitude | 2.28507 _| .47865[ -—-- |---| 1.212 | 3.8708 | 
|17.Excellentteacher | 2.63471_| 45726 | -.38354|  ---_| 1.060 | 4.1496 | 
|18. Excellent course | 2.22667 _| 49763 | 1.238 | 3.8752 | 


Clearly, course motivation (CM) was the most important extraneous variable taken into 
account by adjustments to the Short Form; the stronger the desire of students to take the 
course regardless of who taught it, the more likely high progress ratings would be reported. 
The other two measures of influences beyond the instructor’s control (size of class and 
“other motivation”) did not always have significant regression weights. When they did, 
their weights were negative. If classes were large and/or if “extraneous” student motivation 
(motivation unrelated to a desire for a specific instructor) was low, it was probable that 
progress ratings would be negatively affected, making it necessary to adjust the ratings. 
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To estimate the amount of improvement to Short Form adjustments which might be 
anticipated if the WH item were included, all calculations related to adjustments were 
performed using Diagnostic Form data but omitting Dy and Ey, the measures of extraneous 
influences which would not be available on the Short Form. The amount of variance 
accounted for by extraneous measures (R’) increased from an average of .094 to an average 
of .156, a very substantial improvement (see Appendix C). 
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IV. Reliability 


Classes with 13-17 respondents were used to compute split half reliabilities for each of the 
47 items and for the 5 teaching methods scales described in Section II of this report. Classes 
were randomly divided and means were computed for each half. These means were 
correlated. Results were taken as an estimate of the split half reliability of classes averaging 
7.5 respondents. The Spearman-Brown Prophecy formula® was applied to estimate 
reliabilities for classes averaging 12.5, 24.5, 42.5, and 60 respondents (corresponding to 
class size ranges of 10-14, 15-34, 35-49, and 50+). 


Standard deviations were also computed for each item” or scale and these were used, in 
conjunction with the computed reliabilities, to calculate standard errors of estimate. Results 
are shown in Table 17. 


All measurements include a degree of “error.” The data of Table 17 provide the user with 
information about the likely range within which the “true” mean falls (the theoretical 
average from an infinite number of administrations of the form). In general, the probability 
that the true mean will fall within ? one standard error of the obtained mean is 
approximately two out of three; 95 times in 100 it will fall within two standard errors of the 
obtained mean. 


Txx = ____ ry) 
14+(n-1)ry1 


° Standard deviations were calculated for the 44,447 classes with 10 or more respondents processed between 
1998 and 2001. Items 21-32 (progress ratings) were exceptions to this; for these items, only “relevant” classes 
(those for which the objective was selected as “important” or “essential”) were used in computing standard 
deviations. 
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Table 17 
Reliability and Standard Errors of Items and Scales 
For Four Class Sizes 


All Classes 


10-14 15-34 35-49 50+ 
Teaching Methods 


1. Displayed personal interest in students 
8. 
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2. Helped students answer own questions 
3. Scheduled work helpfully 
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4. Demonstrated imp of subject 


5. Formed teams, discussion groups 
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| 6. Made clearhow topics fit_ | 4.20 | 506 | 
| 3.78 | 570 | . 


7. Explained criticisms 
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Stimulated intellectual effort : 
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9. Encouraged use of multiple resources 3.78 
10. Explained clearly ; 
11. Related to real life 
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13. Introduced stimulating ideas 
14. Involved students in hands on activities 
15. Inspired students to set high goals 
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26. Creative capacities 
27. Broad liberal education 
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81 | 26 | .89 | 
42 | . 82] 24 | .90 | 
| 30 | 89 | 
| 40. Increase positive attitude toward field__| 3.86 | .602 | 75 | 30 | .86 | 
| 41. Excellent instructor | 4.18 | 043 | | .96 | 
| .27 | 89 | 
| Progress on Relevant Objectives (PRO) | 50.9 | 8.6 | .78 3.0 |.92 | 2.4 [95 | 2.0 


*PRO ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10. All 
other ratings were made on a 5-point scale where 1=low and 5=high. 
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Table 17 is continued on the next page. 
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Table 17 (continued) 
Reliability and Standard Errors of Items and Scales 
For Four Class Sizes 


All Classes 


10-14 15-34 49 50+ 


Additional Method Items 
44. Used variety of evaluation methods 9 
45. Expected students to take responsibility 
46. High achievement standards 
47. Used educational technology 
3 
3 
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Teaching Method Scales 


AT3 


Ratings were made on a 5-point scale where 1=low and 5=high. 
Note: Analyses reported in Table 17 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 
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For the five a priori scales, internal consistency reliabilities were computed using 
Cronbach’s Alpha.'° Since inter-correlations of items were generally high (see Table 6), 
these reliabilities were also high, as noted in Table 18. 


Table 18 
Internal Consistency Reliabilities for Teaching Method Scales 


Coefficient Alpha 
Stimulating Student Interest 
Fostering Student Collaboration 


Establishing Rapport 
Encouraging Student Involvement 
Structuring Classroom Experiences 


Note: Analyses reported in Table 18 used a more restricted data set. Classes with response rates less 
than 75% or not reporting the number enrolled were also excluded. 


'0Cronback, L. J. (1951) “Coefficient Alpha and the Internal Structure of Tests,” Psychometrika, 16, 297-334. 
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V. Validity 


What evidence is there that student ratings obtained from the IDEA system can be 
trusted? This section updates previous studies of the system’s validity based on results 
obtained in the most recent three years. Four approaches to validity were taken. 


1. The correlation of student progress ratings and instructor ratings of importance. 
The first study is based on three assumptions: (1) instruction is effective; (2) instructors 


make meaningful and conscientious judgments when they rate the importance of each 
objective; and (3) students make accurate ratings of the progress they make on these 
objectives (the validity question under investigation). If all three assumptions are true, then 
there should be a positive correlation between the instructor’s rating of importance and the 
students’ average rating of progress. To the degree that any of these assumptions is less than 
100% true (instruction is not effective, instructors were not always conscientious in 
identifying objectives, students did not estimate their progress accurately) this correlation 
will be reduced. The correlation will also be attenuated by the fact that importance ratings 
are made using only a 3-point scale. For these reasons, this test of validity is considered to 
be a severe one. 


The bolded numbers in Table 5 provide the information required by this study. The average 
correlation between the instructor’s rating of importance and students’ average rating of 
progress on the corresponding objective across all 12 objectives was +.265. In contrast, the 
average correlation between instructor rating of importance of a given objective and student 
ratings of progress on the other 11 (irrelevant) objectives was +.024. These findings are 
consistent with those reported for other samples dating back to 1973. We conclude that 
students rate their progress on instructional objectives with more than minimal validity. 


2. The consistency of student ratings with intuitive expectations. 
The 20 “methods” items included on the IDEA form were chosen because they have been 


identified as “desirable” or “potent” teaching techniques. Therefore, if student ratings are 
valid, there should be a degree of correspondence between their ratings of progress and their 
perceptions of how frequently the instructor employed these “potent” methods. The data of 
Table 6 make it apparent that the expected correspondence occurred almost uniformly. 


Aside from this expectation of general correspondence, there is the question of whether 
specific correlations make sense. An examination of relevant data in Table 6 shows that 
many intuitive expectations were met. For example, the teaching method most closely 
related to student ratings of progress on “Team skills” (Item 25) was Formed teams or 
“discussion groups” to facilitate learning (Item 5). Progress on “Learning to find and use 
resources for answering questions or solving problems” (Item 29) was most closely related 
to ratings of Encouraged students to use multiple resources to improve understanding (Item 
9). Progress on “Developing a clearer understanding of, and commitment to, personal 
values” (Item 30) was most highly correlated with Asked students to share ideas and 
experiences with others whose backgrounds and viewpoints differ from their own (Item 16). 
Progress ratings on “Developing creative capacities” (Item 26) were most closely related to 
Gave projects, tests, or assignments that required original or creative thinking (Item 19). 


Data provided earlier with respect to the impact of class size on correlations between 
instructional methods and student progress provides additional evidence that student ratings 
were consistent with intuitive expectations (see Table 7). Progress ratings on “Developing 
creative capacities” (Item 26) were substantially related to Formed teams or “discussion 
groups” to facilitate learning (Item 5) for very large classes (where personalized techniques 
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are more problematical), but not for smaller classes. And progress ratings on “Developing a 
clearer understanding of, and commitment to, personal values” (Item 30) was closely related 
to Asked students to help each other understand ideas and concepts (Item 18) if class size 
was less than 35 but was not so useful in larger classes. 


3.The differential validity of the methods items. 
Teaching methods items that were most highly correlated with progress ratings were 


relatively distinctive for each objective (see Table 7). Exceptions were the first two 
objectives (basic cognitive background) and the third and fourth objectives (applications; 
professional skills and viewpoints) where identical lists of “most relevant” teaching 
techniques were identified. But when lists of the eight “most relevant’ methods for “Factual 
knowledge” and “Team skills” were compared, only three were on both. Generally, with 
the exceptions noted above, the amount of overlap between any two sets of “most relevant” 
items was approximately 50 percent. Unless students were making differential judgments in 
answering the questions, such distinctive patterns of relevant teaching methods would not 
have existed. 


4. Correspondence between independently obtained student and faculty ratings. 
Using the Faculty Information Form (see Appendix A) faculty participants are asked to 


respond to a number of questions about the specific class they are teaching. Their answers 
to these questions sometimes suggest how students might rate their progress or otherwise 
evaluate the instructor and class. Several studies were undertaken to determine if these 
expected relationships existed. Their presence would constitute evidence for the validity of 
the system since the instructors and students each made their ratings without knowledge of 
each other’s views. 


In the first of these studies, instructors were asked to rate the impact of various 
circumstances on the learning of students (Contextual Question 4). Circumstances were 
described as having a “Positive,” “In between,” or “Negative” impact on learning. Four of 
them were believed to be especially relevant to overall (global) outcomes: previous 
experience in teaching the course; desire to teach the course; adequacy of students’ 
background and preparation for the course; and student enthusiasm. 


Table 19 compares the average rating on the four global criteria—progress on relevant 
objectives (PRO) and three single-item ratings (increased positive attitude toward the 
subject; excellent teacher; excellent course)—for classes that were rated as having different 
impacts on student learning. PRO results are reported in T Scores, while those for the three 
individual ratings are based on the IDEA system’s 5-point scale. 


In every instance, the expected differences were found. In classes where the circumstance 
was expected to have a positive influence on student learning, global ratings were 
significantly higher than in those where the expected impact was negative. Classes with “in 
between” faculty ratings invariably had “in between” student ratings on these four measures. 
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Table 19 
The Relationship Between Instructor Ratings of Selected Circumstances and Student 
Global Ratings of Teaching and Learning 
Ci Global Rating 
ircumstance/ ae 
Attitude Teacher | Course 

Previously taught 

Desire to teach 

Student background | OSC 
Studententhusiam [OSC 


PRO (Progress on Relevant Objectives) ratings are standardized T Scores. The distribution has a mean of 50 
and standard deviation of 10. All other ratings were made on a 5-point scale where 1=low and 5=high. 
Note: Analyses reported in Table 19 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


A second study focused on the instructor’s description of specific class emphases 
(Contextual Question 3). They indicated whether the class required “None,” “Some,” or 
“Much” of seven activities: writing, oral communication, computer applications, group 
work, mathematical/quantitative work, critical thinking, and creative/artistic/design 
endeavor. If the IDEA system is valid (if both instructor and student ratings can be trusted), 
then there should be a relationship between some of these emphases and progress on related 
objectives. 


Specifically, if “writing” was emphasized, students should report above average progress on 
“Communication skills.” If “critical thinking” was emphasized, above average progress 
should be reported on “Critical analysis.” If “creative/artistic/design endeavor” was 
emphasized, students should report above average progress on “Creative capacities.” And if 
“group work” was emphasized, student progress on “Team skills” should be relatively high. 


Results are shown in Table 20. 
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Table 20 
Relationship Between Instructor Emphasis and 
Relevant Student Progress Ratings 
Instructor Emphasis: Writing 


Student Progress Rating’ 7 None_| Some | Much _| 
Communication Skills 
5360 


Instructor Emphasis: Critical Thinking 
| 66 
1005 


Much 


Team Skills 


“This study used only courses where the learning objective was selected as “important” or “essential,” making 
it a very conservative test of validity. 

Note: Analyses reported in Table 20 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


All four F tests were highly significant (P<.0001). The expected relationships were 
confirmed, thus establishing validity for both instructor and student ratings. 


In a third validity test in which instructor and student ratings were compared the focus was 
on two objectives: Developing specific skills, competencies, and points of view needed by 
professionals in the field most closely related to this course and Gaining a broader 
understanding and appreciation of intellectual/cultural activity (music, science, literature, 
etc.). If the IDEA system is valid, the first of these should be chosen much more frequently 
by those teaching professionally oriented courses (or courses related to the students’ major 
field) while the second should be selected more frequently by instructors teaching courses 
directed to meeting general education or distribution requirements (as indicated by 
Contextual Question 5). 


This expectation was confirmed. More than 78 percent of those teaching professionally 
oriented courses chose the “professional development” objective, compared to 21 percent of 
those teaching general education/distribution courses. On the other hand, over 60 percent of 
the latter chose the “broad liberal education” objective compared to 39 percent of those 
teaching professionally oriented courses. 
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Student progress ratings on these objectives were compared for the two types of classes; 
these comparisons were limited to classes for which the instructor chose the objective in 
question as “relevant.” Results followed a similar pattern. Progress ratings were 
significantly higher on the professional development objective in professionally oriented 
courses (4.15 vs. 3.85 for classes focused on meeting general education/distribution 
requirements). Conversely, the latter averaged 3.72 on the broad liberal education objective 
compared to 3.63 for professionally oriented classes. In both instances, the “t’ test was 
significant beyond the .001 level. 


Since both “relevance” and progress ratings were consistent with those expected if the IDEA 
system were valid, further confirmation of validity was provided. 


A final validity study centered on measures used to adjust student ratings. A number of 
studies have established that students give a much higher priority to courses that prepare 
them for a profession than for those aimed at a general or liberal education. Therefore, those 
teaching courses related to the student’s major interest should receive ratings indicative of 
higher student motivation than those teaching courses designed to meet general education or 
distribution requirements. Relevant measures of motivation are Items 36 and 39 (J had a 
strong desire to take this course; I really wanted to take this course regardless of who was 
teaching it). Results of these two items for five types of classes are given in Table 21. Both 
F tests were significant beyond the .0001 level. 


Table 21 
Motivation Ratings by Principle Type of Student Enrolled in the Class 


36. Strong desire to take | 39. Wanted to take course 
Type of Student this course regardless of who taught it 
| Mean [ sd. | Mean | sd. 


Lower Division, General 334 65 311 55 
Education 


Upper Division, General 3.55 61 3.21 54 
Education 

Lower Division, Specialized | 3.86 | 68 | 3.49 | SS 
Upper Division, Specialized | 3.86 | 60 | 3-44 |S 


Graduate/Professional 


Ratings were made on a 5-point scale where 1=low and 5=high 
Note: Analyses reported in Table 21 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


The IDEA system makes adjustments in ratings to take this type of “extraneous 
circumstance” into account. If adjustments are successful in making the “playing field” 
more even, then they should be positive for those teaching general education courses and 
negative for those teaching courses related to the student’s major. Table 22 provides data to 
test the validity of this expectation (and hence the validity of adjustments). 


All F tests were significant (P<.0001). Without exception, adjustments for classes designed 
to meet general education/distribution requirements at the lower division level were positive, 
ranging from +.02 to +.08 on individual objectives. At the upper division level, adjustments 
for this type of class were generally positive, although small negative figures were obtained 
on 4 of the 12 progress ratings. When pairwise comparisons were made, adjustments for 
upper division general education courses were significantly different (in a positive direction) 
from upper division courses related to the student’s major/professional interests in 15 of the 
16 comparisons. 
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In most comparisons, adjustments for graduate/professional level courses were greater than 
those for the other four types. This was expected since students in such courses are almost 
always highly motivated. The high unadjusted ratings in these courses reflect, in part, this 

motivation! ’. 


Table 22 
Differences Between Adjusted and Unadjusted Ratings 
Among Five Types of Classes 
Type of Class 


General Education/ Specialized/Major 

Criterion Distribution Graduate/ 
Lower Upper Lower Upper | Professional 
Division | Division Division 


fs 6a 


a a 
“Progress on Relevant Objectives ratings are standardized T Scores. The distribution has a mean of 50 and 
standard deviation of 10. All other ratings were made on a 5-point scale where 1=low and 5=high. 
Note: Analyses reported in Table 22 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


Since these results were in line with expectations, it can be concluded that there is validity in 
the IDEA system’s adjustments. 


'! Lower adjusted scores for such classes do not necessarily mean that unadjusted ratings overestimate 
instructional effectiveness. Rather, the quality of instruction is less vital in such classes since high student 
motivation and energy almost ensures high levels of progress. 


D2 


VI. Other Technical Questions 


This section addresses two questions that, while relevant to the interpretation of IDEA 
results, don’t fit into any of the previous five sections. These questions are: 


1. Are results on the Diagnostic and Short Form comparable? 
2. Are there significant differences among disciplines? 


1. Comparability of Diagnostic and Short Forms 


Initially, the two forms were compared by examining the averages for student ratings of 
progress on relevant objectives (those chosen as “Important” or “Essential” by the 
instructor) as well as on the three global ratings of effectiveness (increased positive attitude 
toward the subject, excellence of the teacher, and excellence of the course). Results are 
shown in Table 23. 


Table 23 
Comparison of Ratings on the IDEA Diagnostic Form 
And the IDEA Short Form 


foe 
Objective 
[Communication skills | 18170] 379 | 08 | 
Fincreased positive atiiude | 4447] 386_[ 60] 
PmR827 [435 | 60 


A consistent difference favoring the Short Form is apparent. For the 12 individual 
objectives, these differences averaged .119; for the three global ratings, they averaged .090. 
Differences of this magnitude are significant in both the statistical and the practical sense. 
The practicality of these differences is especially apparent when the distribution of ratings 
on the two forms is examined. See Table 24. 
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Table 24 
Diagnostic and Short Form Distribution of Means of 
Progress Ratings and Global Items (in Percentages) 


Criterion Form’ | <2.00| 2.00- 2.50- 3.00- 3.50- | 4.00- | 4.50+ 
2M) 3a | 300 | 3a | 390 [ao | 

D 0.05 0.34 1.79 8.04 26.68 | 42.28 | 20.83 

S 0.02 0.13 0.95 4.71 20.11 | 43.69 | 30.39 

0.02 0.21 1.20 5.73 20.40 | 41.32 | 31.14 

24. Professional skills, D 0.04 0.36 1.90 8.08 23.44 | 39.18 | 27.00 
PNewpons | 8 [oos | 022 | tar | see | 2051 [4003 | 3156 
[25.Tean sits | § [om | oss | 343 | seo | dose | ar | a196 

0.09 0.95 3.43 8.60 20. 54 34.41 31. 98 

0.59 1.78 5.25 10.69 22.89 | 32.17 | 26.64 
{oar | ost | 311 | 968 | xmas | 3616) 270 | 

27. Broad liberal 0. 75 7. 94 7. 38 15. 09 7A. 68 78. 71 19, 95 
PV eawation———_|_8_[ 020 | ist | 470 | 1260 | 2286 [5268 | 2598 
[28.Communicion sits |S | O26 | isi_| ase | tao | 536 | 3439 | 2013) 

0.26 131 4.52 12.01 25.36 34.39 22.13 
[29.Find,wseresources | § | gon | tot_| 356 | 1305 | 2296 | 3521 | i367 

, S 0.02 1.64 3.56 13.95 32.96 | 35.21 | 13.67 

S 0.10 0.96 4.70 13.31 26.69 | 32.91 | 21.33 

S 0.02 0.58 2.99 10.74 25.48 | 37.25 | 22.94 

S 0.04 0.42 2.93 10.88 30.09 | 37.12 | 18.50 
a ee ee ee 

40. Increased positive D 0.19 1.00 4.42 12.57 27.05 | 34.59 | 20.19 

0.13 0.52 1.86 4.96 13. 24 28, 92 50. 38 
Ferran Ed a Sa A 

0.11 0.67 3.01 9.10 22.47 35.70 28.93 


“D=Diagnostic Form, S=Short Form 


nN 


non 


26. Creative capacities 


nATDadgay 


iw) 


A number of studies were conducted to try to account for these differences. 


One study restricted the comparison of the two forms to classes that were taught by the same 
method (e. g., “Lecture/Discussion,” “Skill/Activity,” etc.). No reduction in differences was 
found for these more homogeneous groups. 


Similar conclusions were drawn when comparisons were restricted to groups of classes that 
were directed to the same audiences (lower division classes for students seeking to meet 
general education or distribution requirements; upper division classes directed to 
specialization interests of students; etc.). The advantage of Short Form users could not be 
accounted for by their tendency to teach different types of students than was true for 
Diagnostic Form users. 
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A special study was made of PRO and the three global ratings at eight institutions that had 
administered approximately equal numbers of both forms in at least 100 classes. Although 
in general the Short Form’s advantage was still apparent, there were some differences 
among institutions. Of the 32 comparisons (4 measures for each of 8 institutions), the Short 
Form mean was higher in 20; but the Diagnostic Form had higher means 7 times, and the 
two were about equal on the other 5 comparisons. 


Disciplinary differences were examined by comparing results on the two forms for the eight 
disciplines where both forms were most commonly used. Differences were relatively small 
in Engineering and Communications departments, but relatively large in Philosophy and 
General Liberal Arts classes. This study was refined by restricting it to the 36 institutions 
that regularly employed both forms. “Within institutional disciplinary differences” were 
similar to those found when disciplinary differences were studied across all institutions. 


The most crucial test was made when the comparison was restricted to the 465 classes taught 
by the same instructor on two occasions—once using the Diagnostic Form and once using 
the Short Form. In this study, only 2 of the 15 comparisons produced significant 
differences; and the magnitude of the significant differences was about .10 less than that 
found in the original studies. 


Finally, the IDEA on-campus coordinators on campuses where substantial use was made of 
both forms were consulted. In most instances, these coordinators reported that the Short 
Form was employed with faculty members whose effectiveness had been well established 
(tenured faculty, others with significant amounts of experience, etc.). In contrast, the 
Diagnostic Form was typically required of junior, temporary, or part-time faculty. 


These reports offered strong support for the view that differences between the two forms 
were artifacts of campus policies that appeared to assure an advantage to the Short Form. 
When coupled with the findings for the “same course, same instructor” study, it was 
concluded that true differences between the two forms were, at most, minor. The decision to 
restrict all normative reporting to the Diagnostic Form meant that norms would reflect the 
full range of faculty users, not a set that represents established, veteran teachers. 


2. Disciplinary differences 


Do results on the IDEA forms differ for different disciplines? This question has been a 
major focus of IDEA’s research program. The short answer is, “Results differ significantly 
across disciplines, and some of these differences are substantial.” The question requires 
relatively complex and detailed analysis. Therefore, it will be addressed in the Center’s next 
technical report. In this report, a sample of disciplinary differences is provided below. 


A minimum of 500 classes was required before a discipline was considered in these 
analyses. A total of 28 disciplines met this standard. Among other matters, the degree to 
which these disciplines identified each objective as “relevant” (“important” or “essential’’) 
was determined. Similarly, for those classes in which the objective was chosen as relevant, 
the average progress rating was computed. These results are summarized below for two of 
the twelve objectives, Creative Capacities and Critical Analysis, in Table 25. 


ay) 


Table 25 
Disciplinary Differences in Relevance and Progress Ratings 
For Two Learning Objectives 
pelt 
% Average % 

Accounting 
|Design/Applied Arts | 69.0 | 401 | 40.4 
[Fine and Applied Arts | 69.0 T4789. 

History 


“Percent identifying objective as “important” or “essential.” 

Ratings were made on a 5-point scale where 1=low and 5=high. 

Note: Analyses reported in Table 25 used a more restricted data set. Classes with response rates less than 75% 
or not reporting the number enrolled were also excluded. 


Instructors indicated that gains in Creative Capacities represented an “Important” or 
“Essential” objective in over half of the classes in Art, Design/Applied Arts, Fine and 
Applied Arts, and Music. In contrast, it was considered “Of no more than minor 
importance” in over 90 percent of the classes in Accounting, Biological/Life Science, 
Chemistry, Economics, Health Professions, Mathematics/Statistics, Physics, and 
Psychology. The average progress rating in relevant (important; essential) classes was much 
higher in disciplines that featured this objective than in those where it was rarely chosen 
(4.21 for disciplines where this objective was popular; 3.13 for those where it was rarely 
chosen). 
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Findings for the Critical Analysis objective were similar. It was considered relevant in over 
two-thirds of the classes in English Literature, History, Liberal Arts/General Studies, and 
Philosophy (where it was rated as relevant in over 93 percent of all classes). But it was rated 
as relevant in fewer than twenty-five percent of the classes in Computer/Information 
Sciences, Foreign Language/Literature, Mathematics/Statistics, and Music. Again, progress 
ratings paralleled these differences, averaging 4.08 for disciplines where it was commonly 
chosen and 3.48 for those where it was infrequently chosen. 


These findings illustrate some of the very large differences among disciplines. Because 


these are so extensive, a full accounting will be delayed until the publication of a subsequent 
technical report. 
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Appendix A 


Faculty Information Form 
Diagnostic Form 
Short Form (used Fall 1998-Summer 2002) 
Short Form (revised Fall 2002) 
Sample IDEA Report (Diagnostic Form) 
Sample IDEA Short Form Report (reflects adjustments described in Appendix C) 
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@yiteA 
CENTER 


Institution: Instructor: 


Faculty Information Form IMPORTANT! 
See Directions to Faculty: 2 aS Improper Marks 
http://ideaedu.org/wp-content/uploads/2015/04/FIFHandout.pdf tt 9 TO oy 


Course Number: Time and Days Class Meets: 


Objectives: Using the scale provided, identify the relevance of each of the twelve objectives to this 
course. As a general rule, prioritize what you want students to learn by selecting no more than 3-5 
objectives as either Important or Essential. The weighting system used to generate the IDEA report 
weighs Essential objectives "2," Important objectives "1," and Minor objectives "0." 

(Scale - M = Minor or No Importance, | = Important, E = Essential) 


mm Last Name (Up to 11 letters) | Init. | 


OO Gaining factual knowledge (terminology, classifications, methods, trends) 
© Learning fundamental principles, generalizations, or theories 
©) Learning to apply course material (to improve thinking, problem solving, and decisions) 
O Developing specific skills, competencies, and points of view needed by professionals in 
the field most closely related to this course 
5. OO Acquiring skills in working with others as a member of a team 
6. OO O Developing creative capacities (writing, inventing, designing, performing in art, music, 
drama, etc.) 
7. QO OO Gaining a broader understanding and appreciation of intellectual/cultural activity (music, 
science, literature, etc.) 
8. ©) O © Developing skill in expressing oneself orally or in writing 
9.) OO Learning how to find and use resources for answering questions or solving problems 
10. QO © Developing a clearer understanding of, and commitment to, personal values 
11. OOO Learning to analyze and critically evaluate ideas, arguments, and points of view 
12. OOO Acauiring an interest in learning more by asking questions and seeking answers 


|A|B|c|D/E/F|G/H| 


© 
OQOOG® 
O@OOG® 
O@OOE® 
O@BOOGE® 
O@OOG® 


® 
© 
® 
® 


Time Class Course 
Begins Number 


®SOOQVO@OOOO®OQOOOO 
®E©OO@OOQGOGGHOQOO® 
=ill 
33 
©OGL©GLOQ] 188 


®SCQOOO@OOHOOO® 
@©©QOQO@QOHOO® 
®@CE©Q@O@2OOO® 
®SEQOWOO@COHGO®S 
®@©C©QOGOO0® 


== Contextual Questions (Research Purposes): 
=== s The IDEA Center will conduct research on these optional questions in order to improve the interpretation of student ratings. 


Which of the following .  {f multiple approaches 

represents the primary are used, which one 

approach to this course? represents the 

(Mark only one) secondary approach? 

(Mark only one) 
(4) = Lecture @) = Lecture 
= Discussion/recitation (2) = Discussion/recitation 

Seminar 
Skill/activity 
Laboratory 


Describe this course in terms of its requirements with respect to 
the features listed below. Use the following code to make your 
responses: 

N = None (or little) required 

S = Some required 
M = Much required 


Se . Writing 
= OO B. Oral communication 

= @ O C. Computer applications 

OO D. Group work 

OO E. Mathematical/quantitative work 
OOO F. Critical thinking 


OOO G. Creative/artistic/design endeavor 


OOOH. Reading 


. Memorization 


Studio 
= Multi-Media 
@) = Practicum/clinic @) = Practicum/clinic 
(0) = Other (0) = Other 
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@ 

® 

6) = 

©) = Field Experience 
@ = 

sg 


Multi-Media 


Contextual Questions Continued: 


Rate each of the circumstances listed below, using the following 5. 
code to respond: 


Please identify the principal type of student 
enrolling in this course 
(Mark only one) 


@ = 


P = Had a positive impact on learning 

| = Neither a positive nor a negative impact 
N = Had a negative impact on learning 

? = Can't judge 


P IN ? 


OQOOOA. 


First-year students/sophomores seeking to 
meet a "general education” or "distribution" 
requirement 

First-year students/sophomores seeking to 
develop background needed for their 
intended specialization 

Upper level non-majors taking the course 
as a "general education" or "distribution“ 
requirement 

Upper level majors (in this or a related 
field of study) seeking competence or 
expertise in their academic/professional 
specialty 

Graduate or professional school students 
Combination of two or more of the above 


types 


Physical facilities and/or equipment 

OOOO B. Your previous experience in teaching this course 

OOOO C. Substantial changes in teaching approach, course 
assignments, content, etc. 

OOOO D. Your desire to teach this course 

Your control over course management decisions 

(objectives, texts, exams, etc.) 

Students’ level of preparation for taking 

the course 

OOOO) G. Students’ level of enthusiasm for the course 

OOOOH. Students’ level of effort to learn 

OOOO I. Technicalinstructional support 


OOOOE 
OOOOF. 


6. Is this class: 


a. Team taught? 
b. Taught through distance learning? 


© Yes 
© Yes 


Discipline Codes (Modified CIP Codes) 


0100 Agricultural Business and Production 9902 Developmental Reading 2700 Mathematics and Statistics 
0200 Agricultural Sciences 9903 Developmental Writing 5009 Music (Performing, Composing, 
Theo! 
0300 Conservation and Renewable Natural 9904 Developmental Natural Sciences ¥) 
Resources , 5116 Nursing 
4506 Economics 
0400 Architecture and Related Programs : 3100 Parks, Recreation, Leisure, and 
; ; 1300 Education Fitness Studies 
0500 Area Ethnic and Cultural Studies : . 
1400 Engineering 3801 Philosophy 
5007 Art (Painting, Drawing, Sculpture) : : ; 
1500 Engineering-Related Technologies 4000 Physical Science (EXCEPT 
3201 Basic Skills i i 
9910 English as Second Language Physles ane. Chemistry) 
2600 Biological Sciences/Life Sciences i 
2301 English Language and Literature SOS eiay sits 
5201 Business, General iti i 
5000 Fine and Applied Arts (EXCEPT 4510 Political Science and Government 
5202 Business Administration and Management Art, Music, and Design and Applied 4200 Psychology 
. . Arts) 
5203 Business - Accounting ; ; 4400 Public Administration and Services 
- ; 1600 Foreign Languages and Literatures (EXCEPT Social Work) 
5208 Business - Finance 
3105 Health and Physical ici i i 
554° Business Jaformation and Data EAUBeiOniEhicce 3900 Religion and Theological Studies 
Processing Services : 4500 Social Sciences (EXCEPT 
, ; 5100 Health Professions and Related Economics, History, Political 
5214 Business - Marketing Sciences (EXCEPT Nursing) Science, and Sociology) 
4005 Chemistry 5199 Health Professions and Related 4407 Social Work and Service 
0900 Communications selences (e-year program) 4511 Sociology 
1100 Computer and Information Sciences GOU8-TIlstory, 2310 Speech and Rhetorical Studies 
4301 Criminal Justice and Corrections PA00 se umen Selences rary ane 
Consumer Sciences Vocational/Technical Programs 
F j see Website: Department codes 
1205 Culinary Arts and Related Services 2400 Liberal Arts & Sciences, General 004000) Pp 
1103 Data Processing Technology (2-year Studies and Humanities 
program) : oy (2y . 9900 Other (to be used when none of the 
2200 General Legal Studies above codes apply) 
5004 Design and Applied Arts (Undergraduate) 
9901 Developmental Math 2500 Library Science 


To see an expanded list of discipline codes go to: http://ideaedu.org/support/discipline-and-department-codes/ 


oe i 

= = FP =SFu DIAGNOSTIC FEEDBACK g | Perea 
- PN (SURVEY FORM - STUDENT REACTIONS TO IMPORTANT! 
ra CENTER| INSTRUCTION AND COURSES) << Swe 
fo Your thoughtful answers to these questions will provide helpful information to your instructor. 

= Describe the frequency of your instructor’s teaching procedures, using the following code: 

= 1=Hardly Ever 2=Occasionally 3=Sometimes 4=Frequently 5=Almost Always 


= 
s 
@ 
5 
® 
a 
zg 
c 
Q 
= 
° 
= 


566666 


OO COOOL OQOOQOOQCQOQOQOOO®D 


il 
eyed 
3 
a 
a 
a 
° 
3 


@ 


wen 23.1) 
wm 24.(1) 
mm 25.04) 
mm 26.(1) 
am 27.(1) 
mm 28.(1) 
mm 29.(1) 
mm 30.(1) 
man 31.4) 
wma 32.4) 


: 
6 


QOQOQOOH OOO®O 
OBO CoOsoOg, OC Oe 


OOOOQOOOOOOOOQOOOOOQOO®8” 


QDOOQOOOO OOOO O9OO0008 


©) Displayed a personal interest in students and their learning 

® Found ways to help students answer their own questions 

© Scheduled course work (class activities, tests, projects) in ways which encouraged students to stay up-to-date in their work 
® Demonstrated the importance and significance of the subject matter 

® Formed “teams" or "discussion groups" to facilitate learning 

©) Made it clear how each topic fit into the course 

©) Explained the reasons for criticisms of students’ academic performance 

6) Stimulated students to intellectual effort beyond that required by most courses 

©) Encouraged students to use multiple resources (e.g. data banks, library holdings, outside experts) to improve understanding 
6) Explained course material clearly and concisely 

(6) Related course material to real life situations 

©) Gave tests, projects, etc. that covered the most important points of the course 

©) Introduced stimulating ideas about the subject 

©) Involved students in "hands on" projects such as research, case studies, or “real life" activities 

©) Inspired students to set and achieve goals which really challenged them 

©) Asked students to share ideas and experiences with others whose backgrounds and viewpoints differ from their own 
©) Provided timely and frequent feedback on tests, reports, projects, etc. to help students improve 

6) Asked students to help each other understand ideas or concepts 

6) Gave projects, tests, or assignments that required original or creative thinking 

6) Encouraged student-faculty interaction outside of class (office visits, phone calls, e-mail, etc.) 


Twelve possible learning objectives are listed below, not. all of which will be relevant in this class. Describe the 
amount of progress you made on each (even those not pursued in this class) by using the following scale: 


1-No apparent progress 


2-Slight progress; | made small gains on this objective. 

3-Moderate progress; | made some gains on this objective. 
4-Substantial progress; | made large gains on this objective. 
5-Exceptional progress; | made outstanding gains on this. objective. 


QBOOOOOOO OOOO 


©) Gaining factual knowledge (terminology, classifications, methods, trends) 

() Learning fundamental principles, generalizations, or theories 

©) Learning to apply course material (to improve thinking, problem solving, and decisions) 

() Developing specific skills, competencies, and points of view needed by professionals in the field most closely 
related to this course 

©) Acquiring skills in working with others as a member of a team 

©) Developing creative capacities (writing, inventing, designing, performing in art, music, drama, etc.) 

6) Gaining a broader understanding and appreciation of intellectual/cultural activity (music, science, literature, etc.) 

6) Developing skill in expressing myself orally or in writing 

©) Learning how to find and use resources for answering questions or solving problems 

G) Developing a clearer understanding of, and commitment to, personal values 

©) Learning to analyze and critically evaluate ideas, arguments, and points of view 

®) Acquiring an interest in learning more by asking my own questions and seeking answers 
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On the next three items, compare this course with others you have taken at this institution, using the following code: 
5=Much More 


1=Much Less than 2=Less than 3=About Average 4=More than 
Most Courses Most Courses Most Courses 


than Most Courses 


The Course: 


33.4) @ @© @_~ © Amount of reading 


34.0 @ @ @~ © Amount of work in other (non-reading) assignments 


35.4) @ © @®_ © Difficulty of subject matter 


Describe your attitudes and behavior in this course, using the following code: 


1=Definitely 2=More False 3=In Between 4=More True 
False Than True Than False 


5=Definitely 
True 


36.4) @ @ @ _~ © thada strong desire to take this course. 

37.4) @ @ @®~ © | worked harder on this course than on most courses | have taken. 

38.4) @ @ @®_~ © | really wanted to take a course from this instructor. 

3.4 @ @ @~ © [really wanted to take this course regardless of who taught it. 

4.4) @ @ @®~ © Asa result of taking this course, | have more positive feelings toward this field of study. 
4.4) @ @ @®_~ ©) Overall, | rate this instructor an excellent teacher. 

42.4 @ @® @ ~ ©) Overall, | rate this course as excellent. 


For the following items, blacken the space which best corresponds to your judgment: 
1=Definitely 2=More False 3=In Between 4=More True 


False Than True Than False 


5=Definitely 
True 


BQ @ @® @ © Asarule, | put forth more effort than other students on academic work. 

44.4) @ @ @®_~ © The instructor used a variety of methods--not only tests--to evaluate student progress on course objectives. 
45.4) @ @ @©@_ ©) The instructor expected students to take their share of responsibility for learning. 

46. @) @ @~ © The instructor had high achievement standards in this class. 

47.4) @ @ @®~ © The instructor used educational technology (e.g., Internet, e-mail, computer exercises, multi-media 


presentations, etc.) to promote learning. 


80 @ © ®© © 80 ©@ © © © 
970 @ © © © 94 © © ®@ © 
0O © © © © 00 ©@ © ®© © 
10 ©@ @©@ © © 1 ©@ © ®©@ © 
20 © © © © 20 ©@ @©@ ®@ © 
30 ©®@ © © © 840 @ © ®© © 
40 © © © © 440 ©@ © @© © 
50 © © © © 5G © © @© © 
50 © © © © 60 ©@ © © © 
70 ©® © © © 70 © © © © 
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Use the space below for comments 
(unless otherwise directed). 

Note: Your written commenis may be 
returned to the instructor, You may want 
to PRINT to protect your anonymity. 


IDeA SHORT FORM - STUDENT REACTIONS TO INSTRUCTION AND COURSES 


@ 


Institution: Instructor: 


Course Number: Time and Days Class Meets: 
(XXX TT OW®BOC® 


Twelve possible learning objectives are listed below. For each, rate your progress in this course compared with your 
progress in other courses you have taken at this college or university. (Of course, ratings on objectives which were 
not addressed by the course will usually be low.) 


In this course, my progress on this objective was: 

1-Low (lowest 10 percent of courses | have taken here) 
2-Low Average (next 20 percent of courses | have taken here) 
3-Average (middle 40 percent of courses | have taken here) 
4-High Average (next 20 percent of courses ! have taken here) 
5-High (highest 10 percent of courses | have taken here) 


IMPORTANT! 


SI TEE... 
E NO.2 PENCIL ONLY - 
SS oro 


mt == 


Progress on: 


1.@ (@) ® @® ® Gaining factual knowledge (terminology, classifications, methods, trends) 
2@ @ @® @® © Learning fundamental principles, generalizations, or theories 
3Q @ @® @®_ © Learming to apply course material (to improve thinking, problem solving, and decisions) 
4Q @® @® @® _ © Developing specific skills, competencies, and points of view needed by professionals in the field most closely 
related to this course 
5.@ @ @® @® _ © Aeauiring skills in working with others as a member of a team 
6.@ @ ® @® 6) Developing creative capacities (writing, inventing, designing, performing in art, music, drama, etc.) 
7.@ @ ® @) ® Gaining a broader understanding and appreciation of intellectual/cultural activity (music, science, literature, etc.) 
8.@) @® ®©® @®© © Developing skill in expressing myself orally or in writing 
9.@ ® ® ® © Learning how to find and use resources for answering questions or solving problems 
mm 10.0) @® ® ® ® Developing a clearer understanding of, and commitment to, personal values 
on 11 .@) @ ® @ ® Learning to analyze and critically evaluate ideas, arguments, and points of view 
mm 12.) @) ® ® ® Acquiring an interest in learning more by asking my own questions and seeking answers 


For the remaining questions, use the following code: 


1=Definitely 2=More False 3=In Between 4=More True 5=Definitely 
False Than True Than False True 


=13.0€) @ @® @® © thadastrong desire to take this course. 

wn 14.0) @® ® @) ® | really wanted to take a course from this instructor. 

mm 15.) @ ® @ ® | really wanted to take this course regardless of who taught it. 

m16.@ @) @) @ ® As a result of taking this course, | have more positive feelings toward this field of study. 

=m 17.4) @) @® @ © Overall, | rate this instructor an excellent teacher. 

=i8.) @ @ @ _ ©® Overall, | rate this course as excellent. 

Ys 

Ys 

— EXTRA QUESTIONS 

—_ if your instructor has extra questions, answer them in the space designated below (questions 19-28). 

= 

=1.0 ©@ © ®@ © 4@ @®@ @®@ @ © Your comments are invited on how the 
=2.0 @©@ © © ® 5Q@ @©@ © © © instructor might improve this course or 
=210 @©@ © ® © 60 © © © © teaching procedures. Use the space 
=220 @ © @®@ © 27.0 ® © @ © provided on the back of this form for 
=230 @ ®©@ ®© © 209 @ © © © your comments. 

=i 


m= Copyright © IDEA Center, 1998 Mark Reflex® by NCS MM98921-2 654 ED06 Printed in U.S.A. Continue on back page 
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= Proper Marks Improper Marks 
IMPORTANT! U8 No. 2 RENE ONLY ig eeeeoee QVBOO@® 


amount of progress you made on each (even those not pursued in this class) by using the following scale: 


1-No apparent progress 


2-Slight progress; | made small gains on this objective. 
3-Moderate progress; | made some gains on this objective. 
4-Substantial progress; | made large gains on this objective. 
5-Exceptional progress; | made outstanding gains on this objective. 


Progress on: 


SHORT FORM - STUDENT REACTIONS TO INSTRUCTION AND COURSES @|IDEA 


Twelve possible learning objectives are listed below, not all of which will be relevant in this class. Describe the 


10 @ @® @®_~ © Gaining factual knowledge (terminology, classifications, methods, trends) 
24) @ @ @~ © Learning fundamental principles, generalizations, or theories 
340 @ @ @®_~ ® Learning to apply course material (to improve thinking, problem solving, and decisions) 
44) @ @® ©®_ © Developing specific skills, competencies, and points of view needed by professionals in the field most closely 
related to this course 
5.@ @ @ @®_~ © Acaguiring skills in working with others as a member of a team 
6. @ @® @®_ © Developing creative capacities (writing, inventing, designing, performing in art, music, drama, etc.) 
7.@Q @ @ @®_~ © Gaining a broader understanding and appreciation of intellectual/cultural activity (music, science, literature, etc.) 
8@Q = @ @® @®_ ©) Developing skill in expressing myself orally or in writing 
9.4 @ @® @®_ © Learning how to find and use resources for answering questions or solving problems 
mmi0.¢) ©) ©) @~ ©) Developing a clearer understanding of, and commitment to, personal values 
m11.4) @ @ @~ © Learning to analyze and critically evaluate ideas, arguments, and points of view 
® © ®© © 


Acquiring an interest in learning more by asking my own questions and seeking answers 


For the remaining questions, use the following code: 
1= Definitely 2=More False 3=In Between 4=More True 5= Definitely 


False Than True Than False True 


As a rule, | put forth more effort than other students on academic work. 

My background prepared me well for this course’s requirements. 

| really wanted to take this course regardless of who taught it. 

As a result of taking this course, | have more positive feelings toward this field of study. 
Overall, | rate this instructor an excellent teacher. 

Overall, | rate this course as excellent. 


! 
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= 
= EXTRA QUESTIONS 
= If your instructor has extra questions, answer them in the space designated below (questions 19-38). 


=1.90 @ © ®© © 40 © © © © 20 ©@ © © © 3% 
=20.0 ©@ © © © 30 @ © © © 320 ©@ © © © 3.@ 
m21.C © © © © 2O © © © © 1.0 ©@ © © © 2% 
m220 ©@ © © © 7.0 @®@ © ®© © 20 @ © © © 3. 
=z. © © © © 20 ©®@ © © © B30 ©@ © © © B® 
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The IDEA Report for SAMPLE, AX S AMPLE DEA 


Business 0230 qu 1:00), Fall 2001-2002 


IDEA UNIVERSITY ENTER 
Number Enrolled: 34 Your results are considered reliable; re-rating by the same students would be unlikely to produce a report 
Number Responding: 29 __ resulting in different conclusion. The percentage of enrollees who provided ratings is high; results can 
85.3% Responding be considered representative of the class as a whole. 
Sections and Purposes of the Report 
Page Section Purpose 
2 I. Overall Measures of Teaching Provides global assessment of teaching effectiveness. Use with 
Effectiveness pages 3 and 6 for administrative use in making personnel 
recommendations. 
3 II. Student Ratings of Progress on Relevant Provides student self-report of learning on objectives identified 
Objectives as relevant (Important or Essential) by the instructor 
4-5 III. Teaching Methods or Style Related to Primarily to help develop a strategy for improving teaching 
Student Ratings of Progress methods (not intended for use in making personnel 
recommendations) 
6 IV. Course Description/Context Primarily to assist in interpreting the results by considering the 


context in which the course was taught 


7-8 V. Statistical Detail Primarily to provide details that may help you or your 
consultants to understand or interpret the report accurately 


Definitions 


Raw Score: Results obtained by using students’ numerical ratings, all of which are based on a scale of 1 (low) to 5 (high). 


Adjusted Score: Ratings that have been statistically adjusted to take into account factors that affect ratings but are beyond the instructor’s control: student 
desire to take the course regardless of who taught it (item #39); student work habits (item #43); instructor reported class size; student effort not 
attributable to the instructor (multiple items); and course difficulty not attributable to the instructor (multiple items). 


T Score: A statistically derived score that makes it easy to compare various measures. Unlike raw scores, which have different averages and standard 
deviations (variabilities), T Scores all have an average of 50 and a standard deviation of 10. This means that 40% of all T Scores will be in the range 
of 45-55, while less than 2% will be below 30 or above 70. 


Similar Classes: On Page 4, ratings of specific teaching methods are compared with national averages for classes of "similar size and level of student 
motivation." Your ratings are compared with those from one of 20 groups defined by considering both class size (less than 15; 15-34; 35-49; 50 or 
more) and average student response to item 39-/ really wanted to take this course regardless of who taught it (under 2.62; 2.62-3.05; 3.06-3.63; 
3.64-4.08; 4.09 or above). 


Understanding the Graphs 
Most results are presented on graphs. Unadjusted T Scores are shown by the symbol X ; adjusted T Scores are shown by the 
symbol @. In most cases, we use a line on both sides of a symbol to indicate that ratings have a "margin of error"; the line 
represents + one standard error of measurement, a statistical indication of the reliability of the measure. 


A Few Words of Caution 
1. Normative information and the process for adjusting scores were updated using classes rated during the 1998-99, 1999-2000, and 2000-2001 academic 
years. Exercise caution when comparing T Scores and adjusted scores with those for classes processed prior to December 1, 2001. The new norms 
have slightly higher item averages. Therefore, T Scores for a given average will be somewhat lower than those for past years. If results are being 
summarized with classes processed prior to December 1, 2001, review both T Scores and raw scores to determine if differences are due to a more 
competitive normative group or if the item averages have actually changed. 


2. Student ratings can make a useful contribution to the appraisal of teaching effectiveness and to the development of improvement strategies. However, they 
have distinct limitations which need to be acknowledged before appropriate use can be made of them. Please read the Overview of Student Ratings: 
Value and Limitations (www.idea.ksu.edu). 


Faculty Name: SAMPLE, AX Term: Fall 2001-2002 
Course: Business 0230 Page 2 


Section I. Overall Measures of Teaching Effectiveness 


This section compares your results with those for other instructors and courses in the national database on four OVERALL 
MEASURES OF TEACHING EFFECTIVENESS. The primary value of this information is to aid in making administrative 
recommendations; if this is the only use you will make of the report, you need to consult only these results along with page 3 and 
the context provided by Section IV, page 6. Please remember that most of the classes included in the database have been taught in 
a reasonably successful manner; therefore, a rating which is "below average" does not necessarily mean that the quality of 
instruction was unacceptable. Additional sources of evidence should always be used to review teaching effectiveness. 


40% of all Your Average * 
Overall Measures of (5-Point Scale) 


IDEA 
: Unadj. || 2% ofall ‘ 2% of all Average 
Effectiveness Adj. classes classes Adjusted 


1. Progress on Relevant (Essential 47 
and Important) Objectives 51 
2. Improved Student Attitude 


3. Overall Excellence of Teacher 
0 


4. Overall Excellence of Course 


--><4 Unadjusted T Score + one standard error of measurement 

[+-®+i Adjusted T Score + one standard error of measurement: adjusted for student work habits (item #43); student desire to take the course regardless of who 
taught it (item #39); instructor reported class size; student effort not attributable to the instructor (multiple items); and course difficulty not attributable 
to the instructor (multiple items). 


20 30 40 45 50 55 60 70 8 
T Score--Comparison with the IDEA Database ** 


You may wish to assign these ratings to categories like those that have been used historically with the IDEA system. Simply assign 
T Scores to categories as follows: Low (lowest 10%)=T Score below 37; Low Average (next 20%)=T Score 37-44; Average 
(middle 40%)=T Score 45-55; High Average (next 20%)=T Score 56-63; and High (highest 10%)=T Score above 63. 


a 


1. Progress on Relevant (Essential and Important) Objectives. Because student learning is the central purpose of teaching, and 
because you chose the objectives considered by this measure, this is probably the most vital measure of effectiveness. A double 
weight is given to student ratings of progress on objectives you chose as Essential, and a single weight to those chosen as Important 
; objectives identified as being of Minor or No Importance were ignored in developing this measure. 


2. Improved Student Attitude. This shows the average response of students to item 40, "As a result of taking this course, I have 
more positive feelings toward this field of study." This rating may be most meaningful for courses that are taken by many 
non-majors. Most teachers hope that such students will develop a respect and appreciation for the discipline even if they choose to 
take no additional courses in it. The IDEA national average for this item is 3.9. 


3. Overall Excellence of Teacher. This shows the average response to item 41, "Overall, J rate this instructor an excellent teacher." 
Overall impressions of a teacher affect student attitudes, effort, and learning. The IDEA national average for this item is 4.2. 


4. Overall Excellence of Course. This shows the average response to item 42, "Overall, I rate this course as excellent." This 
evaluation is likely determined by a number of factors (e.g., teaching style, student satisfaction with course outcomes, and 
characteristics such as organization, selection of readings, and/or other influences). The IDEA national average for this item is 3.9. 


i 


NA,: Based on a combination of ratings where an average on a 5-point scale is not comparable. 


* Statistically, adjustments can exceed 5.0 on the 5-point scale. If this occurs, "Your Average," reported in the table above, will be rounded to 5.0. However, 
the T Score reported will reflect the actual adjusted score, which may exceed 5.0. Therefore, identical adjusted scores of 5.0 may have different adjusted T 
Scores. 


** Normative information (T Scores) and the process for adjusting scores were updated on December 1, 2001. See page | for "A Few Words of Caution." 


Faculty Name: SAMPLE, AX 
Course: Business 0230 


Section 


Term: Fall 2001-2002 
Page 3 
Il. Student Ratings of Progress on Relevant Objectives 


This graph shows student progress ratings on the objectives you chose as Essential (Part A) and those you chose as Important 
(Part B). To the degree that students make progress on the objectives you emphasize, your teaching has been effective. 


Unadj. 
Adj. 
44 
21. Factual knowledge 47 
- ‘ 45 
22. Principles and theories 49 


Part B. Important Objectives 


23. Apply course material z 
31. Analysis and critical evaluation Z 
2 


Part A. Essential Objectives 


40% of all Your Average * 


(5-Point Scale) 


0 


8 


IDEA 
Average 


2% of all 
classes 


2% of all 
classes 


0 30 40 45 50 55 60 
T Score--Comparison with the IDEA Database where the 


Objective was Selected as "Essential" or “Important” ** 


-3€H Unadjusted T Score + one standard error of measurement 


/-@H1 Adjusted T Score + one standard error of 
taught it (item #39); instructor reported cl 
to the instructor (multiple items). 


measurement: adjusted for student work habits (item #43); student desire to take the course regardless of who 
ass size; student effort not attributable to the instructor (multiple items); and course difficulty not attributable 


Similar to Section 1, you may wish to assign ratings to categories. Simply assign T Scores to categories as follows: Low (lowest 
10%)=T Score below 37; Low Average (next 20%)=T Score 37-44; Average (middle 40%)=T Score 45-55; High Average (next 


20%)=T Score 56-63; and High (highest 


10%)=T Score above 63. 


It is recommended that priority attention be given to Essential objectives with progress ratings that are below average. The second 
priority might be directed to Important objectives for which progress ratings are below average. A third priority might be Essential 


or Important objective 
suggested that your present methods of te 
needed in order to ensure that your teachi 


s for which progress ratings are in the average range. If all progress ratings are above the average range, it is 


aching are effective and changes in your teaching style or approaches do not appear to be 
ng promotes student learning. If improvement is needed, strategies can be formulated by 


examining teaching methods or style associated with progress ratings on the objectives chosen for priority attention. These are 
identified in Section Il] (pages 4 and 5) of this report. 


Note: Students in your class also rated their progress on the objectives that you classified as being of Minor or No Importance. 


These ratings are considered irrelevant in 
objectives, found in Section V (Statistical 
"additional" effects of your instruction. 


judging your teaching effectiveness. However, a review of student ratings on these 
Detail pages 7 and 8), may provide you with insights about some "unintended" or 


NN 


* Statistically, adjustments can exceed 5.0 on the 5 


the T Score reported will reflect the actual adjuste 
Scores. 


-point scale. If this occurs, "Your Average," reported in the table above, will be rounded to 5.0. However, 
d score, which may exceed 5.0. Therefore, identical adjusted scores of 5.0 may have different adjusted T 


** Normative information (T Scores) and the process for adjusting scores were updated on December 1, 2001. See page 1 for"A Few Words of Caution.” 
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Section III. Teaching Methods or Style Related to Student Ratings of Progress 
ne 
This section focuses on specific teaching methods. Results are given in three parts. Part One graphically compares ratings of your 
teaching methods with those of others who teach classes similar to this one in terms of size and level of student motivation (see 
page 1). Part Two identifies the teaching methods most closely related to attaining your Important and Essential objectives, 
providing a basis for developing improvement strategies. Part Three highlights potential areas to emphasize for improvement 
efforts and teaching strengths that should be retained. 


Part One: The graphs below show methods that were more frequently used (your ratings were at least 0.3 above average for classes 
of similar size and level of student motivation) and those that were Jess frequently used (your ratings were at least 0.3 below the 
average of such classes). Not all teaching methods promote progress on every learning objective. The methods that are especially 
relevant to each of your Essential and Important objectives are identified in Part Two (page 5). 


Comparison with Classes of Similar 
Teaching Methods and Style Your Average Rating Size and Level of Student Motivation 


Less Frequent More Frequent 


A. Stimulating Student Interest (Mean of 4, 8, 13, 15) 


4. Demonstrated the importance and significance of the subject 
matter 


8. Stimulated students to intellectual effort beyond that required by 
most courses 


13. Introduced stimulating ideas about the subject 


15. Inspired students to set and achieve goals which really challenged 
them 


B. Fostering Student Collaboration (Mean of 5, 16, 18) 
5. Formed "teams" or "discussion groups" to facilitate learning 


16. Asked students to share ideas and experiences with others whose 
backgrounds and viewpoints differ from their own 


18. Asked students to help each other understand ideas or concepts 


C. Establishing Rapport (Mean of 1, 2, 7, 20) 
1. Displayed a personal interest in students and their learning 
2. Found ways to help students answer their own questions 


7. Explained the reasons for criticisms of students’ academic 
performance 


20. Encouraged student-faculty interaction outside of class (office 
visits, phone calls, e-mail, etc.) 


D. Encouraging Student Involvement (Mean of 9, 11, 14, 19) 


9. Encouraged students to use multiple resources (e.g. data banks, 
library holdings, outside experts) to improve understanding 


11. Related course material to real life situations 


14. Involved students in "hands on" projects such as research, case 
studies, or “real life" activities 


19. Gave projects, tests, or assignments that required original or 
creative thinking 


E. Structuring Classroom Experiences (Mean of 3, 6, 10, 12, 17) 


3. Scheduled course work (class activities, tests, projects) in ways 
which encouraged students to stay up-to-date in their work 


6. Made it clear how each topic fit into the course 
10. Explained course material clearly and concisely 


12. Gave tests, projects, etc. that covered the most important points of 
the course 


17. Provided timely and frequent feedback on tests, reports, projects, 
etc. to help students improve 


1.0 2.0 3.0 4.0 5.0 -0.75 0.3 0.3 0.75 
-o4 Average category rating + one standard error of measurement +*®4 Average item tating + one standard error of measurement 


Faculty Name: SAMPLE, AX Term: Fall 2001-2002 
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Section III. Teaching Methods or Style Related to Student Ratings of Progress (continued) 


Part Two: Column 1 below again lists those objectives you listed as Essential or Important. Column 2 lists those teaching methods 
which in combination are most closely related to progress ratings on your chosen objectives. Column 3 separates out those teaching 
methods that you used more or less frequently than those teaching classes similar size and motivation. (The numbers in Columns 2 
and 3 refer to the teaching methods numbered 1-20 on the graphical presentations in Part One, page 4.) 


Column 1 Column 2 Column 3 
Chosen Objectives Most Relevant Teaching Methods * Relevant Methods to be Considered 
Strengths Consider Increasing 
to Retain Frequency 
Essential Objectives 
21. Factual knowledge 2,4,6,8,10,12,13,15 10,12 
22. Principles and theories 2,4,6,8,10,12,13,15 10,12 
Important Objectives 
23. Apply course material 1,2,4,6,7,8,10,11,13,15 10.11. 
31. Analysis and critical evaluation 2,7,8,13,15,16,18,19 16,18 


a 


Part Three: This section summarizes teaching methods to consider for improvement strategies and methods which are effective and 
should be retained. 


Potential Areas to Consider Increasing Frequency of Use 
Generally, improvement efforts are most successful if they focus on no more than three teaching strategies at a time. These results 
suggest that your improvement strategies might best be chosen from the following teaching methods: 
16. Asked students to share ideas and experiences with others whose backgrounds and viewpoints differ from their own 
18. Asked students to help each other understand ideas or concepts 


Strengths to Retain 

In doing so, you should take care to retain the methods which are currently effective, including: 
10. Explained course material clearly and concisely 
12. Gave tests, projects, etc. that covered the most important points of the course 
11. Related course material to real life situations 


a 
* Underlined item numbers are highly correlated with the learning objective (.60 or above). Others are moderately correlated (.50-.59). 
See The IDEA Center’s homepage for more information (www.idea.ksu.edu). 
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Section IV. Course Description/Context 


em 


This section describes several aspects of your course. Some of the description summarizes information you supplied when you 
administered the IDEA form, and some of the information comes from student responses. Information on this page provides the 
context in which the class was taught, which should guide the interpretation of the ratings. The IDEA Center will conduct 
additional research on these data to determine more precisely how they can improve interpretation of the report. 


le 


Instructor Ratings 


Course Description: 


Primary Instructional Type: Lecture Team Taught: No 
Secondary Instructional Type: Other/Not Indicated Distance Learning: No 
Principal Type of Student: Underclassmen, majors 


Special Circumstances: 


Positive Impact on Learning Neither Positive nor Negative Impact Negative Impact on Learning 
Previous experience teaching course Physical facilities and/or equipment Adequacy of students’ 
Desire to teach course Changes in teaching approach background/preparation 
Control over course management Student enthusiasm 

decisions Student effort 


Technical/instructional support 


Course Requirements: 


Much Required Some Required None (or little) Required 
Mathematical/quantitative work Writing 
Critical thinking Oral communication 
Computer applications 
Group work 


Creative/artistic/design endeavor 


pees of me ee: 
— T Score 
1 


12 14 


Student Ratings 


Course Description 


33. Amount of reading 

34. Amount of work in other (non-reading) 
assignments 

35. Difficulty of subject matter 


*1 = Much less than most courses 2=Less than most courses 3=Aboutaverage 4=More than mostcourses 5 = Much more than most courses 


aa > ee si ca ae 
Average T Score 


Similar to Sections I and II, you may wish to assign ratings to categories. Simply assign T Scores to categories as follows: Low 
(lowest 10%)=T Score below 37; Low Average (next 20%)=T Score 37-44; Average (middle 40%)=T Score 45-55; High Average 
(next 20%)=T Score 56-63; and High (highest 10%)=T Score above 63. 


_— 
= 


Self-Ratings 


37, I worked harder on this course than on most 
courses I have taken. 

39. I really wanted to take this course regardless of 
who taught it. 

43. As a rule, I put forth more effort than other 

students on academic work. 


**] = Definitely false 2= More false than true 3=Inbetween 4= More true than false 5= Definitely true 
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Section V. Statistical Detail: Item Frequencies, Averages, and Standard Deviations 


Items 1-20: Teaching Methods Items 21-32: Progress on Objectives 


Key: 1=Hardly Ever 2=Occasionally 3=Sometimes Key: 1=Low 2=Low Average 3=Average 
4=Frequentl 5=Almost Always 4=High Average 5=High 


N 


oO 
—) 
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Bold items were selected as Essential or Important. 


— — 
Oy. 


Items 33-35: The Course 


Key: 1=Much Less than Most Courses 2=Less than Most Courses 
3=About Average 4=More than Most Courses 
5=Much More than Most Courses 


—>— — bo 
oo NS > 
Ooroooercjcodooooooeogg9gpneg°9ocoerFr oO 2 


N 
° 


Items 44-47: Experimental Items 36-43: Self and Global Outcomes 
Key: 1=Definitely False 2=More False Than True Key: 1=Definitely False 2=More False Than True 
3=In Between 4=More True Than False 3=In Between 4=More True Than False 


5=Definitely True 5=Definitely True 
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Section V. Statistical Detail: Continued 


Items 48-66: Extra Questions 


es 


58. 29 |N/A N/A 

59. 
60. 
6l. 
62. 
63. 
64. 
65. 
66. 


19 
49. 18 
50. 19 
51. 15 


29 |N/A N/A 


52. 11 
53. 10 
54. 
20; 
56. 
57. 
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0 
0 
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The IDEA Short Form Report for SAMPLE, SF 
Design & Applied Arts 0420 (Tu 9:30 ), Fall 2002-2003 S i\ Mi DIL 
IDEA UNIVERSITY 
Local Code: 0113 


Number Enrolled: 18 Your results are considered fairly reliable; it is unlikely that re-rating by the same students would 
Number Responding: 15 _ produce more than a moderate change in your report. The percentage of enrollees who provided ratings 
83.3 % Responding is high; results can be considered representative of the class as a whole. 


Sections and Purposes of the Report 


Page Section Purpose 
2 I. Overall Measures of Teaching Provides global assessment of teaching effectiveness. Use 
Effectiveness with pages 3 and 4 for administrative use in making personnel 
recommendations. 
3 II. Student Ratings of Progress on Relevant Provides student self-report of learning on objectives 
Objectives identified as relevant (Important or Essential) by the instructor 
4 Il. Course Description/Context Primarily to assist in interpreting the results by considering 


the context in which the course was taught 


4 IV. Statistical Detail Primarily to provide details which may help you or your 
consultants to understand or interpret the report accurately 


Definitions 


Raw Score: Results obtained by using students’ numerical ratings, all of which are based on a scale of 1 (low) to 5 (high). 


Adjusted Score: Ratings have been statistically adjusted to take into account factors that affect ratings but are not under the instructor's control: student 


work habits (item #13); student desire to take the course regardless of who taught it (item #15); and instructor reported class size. 


T Score: A statistically derived score that makes it easy to compare various measures. Unlike raw scores which have different averages and standard 
deviations (variabilities), T Scores all have an average of 5Q and a standard deviation of 10. This means that 40% of all T Scores will be in the range of 
45-55, while less than 2% will be below 30 or above 70. 


Understanding the Graphs 
Most results are presented on graphs. Unadjusted T Scores are shown by the symbol X ; adjusted T Scores are shown by the symbol 
@ . In most cases, we use a line on both sides of a symbol to indicate that ratings have a “margin of error"; the line represents + one 


standard error of measurement, a statistical indication of the reliability of the measure. 


A Few Words of Caution 
1. Normative information was updated using classes rated during the 1998-99, 1999-2000, and 2000-2001 academic years. 


Exercise caution when comparing T Scores with those for classes processed prior to December 1, 2001. The new 
norms have slightly higher item averages. Therefore, T Scores for a given average will be somewhat lower than those for past 
years. If results are being summarized with classes processed prior to December 1, 2001, review both T Scores and raw scores 
to determine if differences are due to a more competitive normative group or if the item averages have actually changed. 


2. The process for adjusting scores was updated on October 7, 2002. Use caution when comparing adjusted scores with classes 
processed prior to that date. 
3. Student ratings can make a useful contribution to the appraisal of teaching effectiveness and to the development of 


improvement strategies. However, they have distinct limitations that need to be acknowledged before appropriate use can be 
made of them. Please read Overview of Student Ratings: Value and Limitations. (www.idea.ksu.edu) 
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Section I. Overall Measures of Teaching Effectiveness 


This section compares your results with those for other instructors and courses in the national database on four OVERALL 
MEASURES OF TEACHING EFFECTIVENESS. The primary value of this information is to aid in making administrative 
recommendations; if this is the only use you will make of the report, you need to consult only these results along with page 
3 and the context provided by Section III, page 4. Please remember that most of the classes included in the database have been 
taught in a reasonably successful manner; therefore, a rating which is "below average" does not necessarily mean that the quality of 
instruction was unacceptable. Additional sources of evidence should always be used to review teaching effectiveness. 


40% of all Your Average * | IDEA 
Overall Measure of 7 : (5-Point Scale) | Average 


Effectiveness | Raw [Adjusted 


20 30 40 45 50 55 60 70 80 
T Score--Comparison with the IDEA Database ** 


-3<41 Unadjusted T Score + one standard error of measurement 
--@ + Adjusted T Score + one standard error of measurement: adjusted for student work habits (item #13), student desire to take the course regardless of who 


taught it (item #15); and instructor reported class size. 


You may wish to assign these ratings to categories like those that have been used historically with the IDEA system. Simply assign T Scores to categories as 
follows: Low (lowest 10%)=T Score below 37; Low Average (next 20%)=T Score 37-44; Average (middle 40%)=T Score 45-55; High Average (next 
20%)=T Score 56-63; and High (highest 10%)=T Score above 63. 


1. Progress on Relevant (Essential and Important) Objectives. Because student learning is the central purpose of teaching, and 
because you chose the objectives considered by this measure, this is probably the most vital measure of effectiveness. A double 
weight is given to student ratings of progress on objectives you chose as Essential, and a single weight to those chosen as 
Important, objectives identified as being of Minor or No Importance were ignored in developing this measure. 


2. Improved Student Attitude. The graph shows the average response of students to item 16, "As a result of taking this course, I 
have more positive feelings toward this field of study." This rating is most meaningful for courses that are taken by many 
non-majors. Most teachers hope that such students will develop a respect and appreciation for the discipline even if they choose to 
take no additional courses in it. The IDEA national average for this item is 3.9. 


3. Overall Excellence of Teacher. This shows the average response to item 17, "Overall, I rate this instructor an excellent 
teacher." Overall impressions of a teacher affect student attitudes, effort, and learning. The IDEA national average for this item 


is 4.2. 


4. Overall Excellence of Course. This shows the average response to item 18, "Overall, I rate this course as excellent." This 
evaluation is likely determined by a number of factors (e.g., teaching style, student satisfaction with course outcomes, and 
characteristics such as organization, selection of readings and/or other influences). The IDEA national average for this item is 3.9. 


NA,: Based on a combination of ratings where an average on a 5-point scale is not comparable. 


* Statistically, adjustments can exceed 5.0 on the 5-point scale. If this occurs, “Your Average," reported in the table above, will be rounded to 5.0. However, 
the T Score reported will reflect the actual adjusted score, which may exceed 5.0. Therefore, identical adjusted scores of 5.0 may have different adjusted T 
Scores. 


** Normative information (T Scores) was updated on December 1, 2001. See page 1 for "A Few Words of Caution.” 
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Section II. Student Ratings of Progress on Relevant Objectives 


This graph shows student progress ratings on the objectives you chose as Essential (Part A) and those you chose as Important (Part 
B). To the degree that students make progress on the objectives you stress, your teaching has been effective. 


40% of all Your Average * | IDEA 
Part A. Essential Objectives | Unadj. | 2% of all (5-Point Scale) | Average 
Ad. | os Adjusted 


Part B. Important Objectives 


4, Professional skills, viewpoints 2 } 
0 


20 30 40 45 50 55 60 70 8 
T Score--Comparison with the IDEA Database where the 
Objective was Selected as "Essential" or "Important" ** 


+ ><—| Unadjusted T Score + one standard error of measurement 
/-@—1 Adjusted T Score + one standard error of measurement: adjusted for student work habits (item #13); student desire to take the course regardless of who 


taught it (item #15); and instructor reported class size. 


Similar to Section I, you may wish to assign ratings to categories. Simply assign T Scores to categories as follows: Low (lowest 
10%)=T Score below 37; Low Average (next 20%)=T Score 37-44; Average (middle 40%)=T Score 45-55; High Average (next 
20%)=T Score 56-63; and High (highest 10%)=T Score above 63. 


These graphs are intended to help you identify a focus for improving your instructional effectiveness. If student progress ratings on 
Important or Essential objectives are disappointing, you are encouraged to discuss improvement strategies with your department 
head, the campus faculty development specialist, or a colleague. Such strategies could focus on matters such as teaching 
methods/styles, class activities and assignments, the text and other readings, assessment/feedback, and the need for course 
pre-requisites. You might also consider using the IDEA Long Form the next time you solicit student ratings, since it is designed to 
help identify specific teaching methods to use in improvement efforts. 


Note: Students in your class also rated their progress on the objectives that you classified as being of Minor or No Importance. 
These ratings are considered irrelevant in judging your teaching effectiveness. However, a review of student ratings on these 
objectives, found in Section IV (Statistical Detail), may provide you with insights about some "unintended" or "additional" effects 


of your instruction. 


a 
* Statistically, adjustments can exceed 5.0 on the 5-point scale. If this occurs, "Your Average,” reported in the table above, will be rounded to 5.0. However, 

the T Score reported will reflect the actual adjusted score, which may exceed 5.0. Therefore, identical adjusted scores of 5.0 may have different adjusted T 
Scores. 

** Nommative information (T Scores) was updated on December 1, 2001. See page 1 for "A Few Words of Caution." 
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Section III. Course Description/Context 


This section describes several aspects of your course. This description summarizes information you supplied when you administered 
the IDEA form. Information on this page provides the context in which the class was taught, which should guide the interpretation 
of the ratings. The IDEA Center will conduct additional research on these data to determine more precisely how they can improve 
interpretation of the report. 


Course Description: 
Primary Instructional Type: Skill/activity Team Taught: No Distance Learning: No 
Secondary Instructional Type: Other/Not Indicated Principal Type of Student: Upperclassmen, majors 


Instructor’s Ratings of Special Circumstances: 
Positive Impact on Learning: Previous experience teaching course, Control over course management decisions 
Neither Positive nor Negative Impact: Physical facilities and/or equipment, Changes in teaching approach, Desire to teach course 
, Adequacy of students’ background/preparation, Student enthusiasm, Student effort, 
Technical/instructional support 
Negative Impact on Learning: 


Instructor’s Ratings of Course Requirements: 
Much Required: Creative/artistic/design endeavor 
Some Required: Group work, Critical thinking 
None (or little) Required: Writing, Oral communication, Computer applications, Mathematical/quantitative work 


Section IV. Statistical Detail: Item Frequencies, Averages, and Standard Deviations 


Items 1-12: Progress on Objectives Items 13-18: Self-Ratings 

Key: 1=Low 2=Low Average 3=Average 4=High Average Key: 1=Definitely False 2=More False Than True 

S=Hig 3=In Between 4=More True Than False 
1 5=Definitely True 


2 3 4 5 Omit 

i; 0 0 3 35 i 0 
2. 0 0 3 5 7 0 
3 0 2 4 4 5 0 
4. 0 2 4 2 6 1 
5; 1 1 8 4 1 0 
6. 1 2 4 3 5 0 
7. 0 1 3 6 5 0 
8. 1 2 6 3 3 0 
2; 0 1 5 6 3 0 
10. 1 1 4 4 5 0 
11. 1 0 4 4 6 0 

5 4 0 


Bold items were selected as Essential or Important. 


File: REVISEDSF.SFR. 
Class ID: 100075 November 14, 2002 


Appendix B 


Calculating Scores Reported in The IDEA Report 
(Diagnostic Form) for Individual Faculty Members 


Appendix A includes a sample of the report participants receive for each class. The figures 
on this report were computer-generated. For those who would like to calculate these figures 
by hand, either to check their accuracy or to get a better feel for what goes into a given 
calculation, Appendix B describes the process that is followed in making calculations. 
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I. Necessary Raw Data 


A. National data base results for progress ratings on “relevant” classes (those in which the 


instructor identified the objective as “Important” or “Essential” ). 


Objective Mean s.d. R’ 
21. Gaining factual knowledge. . . 4.0013 494 .1761 
22. Learning principles, theories. . . 3.9443 A485 1633 
23. Applying course material. . . 3.9874 516 = .2248 
24. Developing professional skills, competency. . . 4.0420 524 = .2380 
25. Acquiring team skills. . . 3.9285 632 .1611 
26. Developing creative capacities. . . 3.8668 701 = .1940 
27. Gaining a broad liberal education. . . 3.6948 732.1648 
28. Developing communication skills. . . 3.7887 676 .1930 
29. Learning to find and use necessary recourses. . . 371322 71.1687 
30. Values development, clarification. . . 37119 629.1599 
31. Learning to critically evaluate... . 3.8438 589 =—.1186 
32. Acquiring interest in learning more. . . 3.7907 561 = .2056 
B. Means and standard deviations of ratings on three “global outcomes” measures. 

Mean s.d.  _R?_ 
AO. Increased positive feelings toward subject. . . 3.8611 .602 .3606 
41. Overall, instructor was excellent. . . 4.1815 642 .0883 
42. Overall, course was excellent. . . 3.9198 .607 = .2938 
C. Information from statistical detail (Section V, page 7, of IDEA Report) 

Mean 

Progress on Essential Objectives Reported Calculated 
21. Factual knowledge. . . Sut 3.7241 
22. Principles, theories. . . Se 3.7241 
Progress on Important Objectives 
23. Applying course material. . . 4.1 4.0690 
31. Learning to critically evaluate. . . 59. 3.9310 
Global Ratings 
40. Increased positive feelings toward subject. . . Fe 50552 
41, Overall, instructor was excellent. . . 4.6 4.5517 
42. Overall, course was excellent. . . 3.9 3.8966 
Items Needed to Make Adjustments 
39. Course motivation. . . 3.1 3.1034 
43. Work habits. . . 3.3 3.3448 
Number enrolled (page 1 of IDEA Report) 34 -- 
8. Stimulated high intellectual effort. . . 3.8 3.7931 
33. Amount of reading. . . 3.1 3.1034 
34. Amount of other work. . . 3.4 323929 
35. Difficulty of course. . . 3.6 3.6429 
37. Effort (worked harder than normal. . .) 3.7 3.6897 
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II. Preliminary Calculations 


A. Calculating Dy 
Dy = Mean of Item 35 minus Predicted Mean of Item 35; or Dy = X35 — Pred X35 
Pred X35 = .13412 Xg + .23986 X33 + .40303 X34 +.74331 (Technical Report , p. 37) 
5097 + .7436 + 1.3703 +.7433 = 3.3669 


Dy = 3.6429 — 3.3669 = .2760 


B. Calculating Ey 
En = X37 — Pred X37 
Pred X37 = .35690 Xg + .11142 X33 + .51595 X34 + .06562 (Tech Report, p. 37) 
= 1.3562 + .3454 + 1.7542 + .0656 = 3.5214 
En = 3.6897 — 3.5214 = .1683 
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III. Calculating Adjusted Scores (from formulas on p. 38 of Technical Report) 


Adjusted progress rating on Item 21, Gaining factual knowledge. . . 
Predicted X91 = .27568 X39 + .38141 X43 + .09434 Dy -.0722 Ey +1.69981 
= .8555 +1.2757 +.0260 -.0122 +1.6998 = 3.8448 
Residual = X); — Pred. X2;= 3.7241-3.8448 = -.1207 
Adjusted X21 = Grand Mean, Item 21 +(Residual)(1 + R’) 
= 4.0013 + (-1207)(1.1761) = 4.0013 - .1420 = 3.8593 (IDEA Report, p. 3) 


Adjusted progress rating on Item 22, Learning principles and theories. . . 
Predicted X22 = .25225X39+.39835 X43-.001N+.09683Dy-.1244En+ 1.67488 
= .7828 +1.3324 -.0340+.0267 -.0209 +1.6750 = 3.7620 
Residual = X92 — Pred X22 = 3.7241 — 3.7620 = -.0379 
Adjusted X22 = Grand Mean, Item 22 + (Residual)(1 + R’) = 
3.9443 + (-.0379)(1.1633) = 3.9443 - .0441 = 3.9002 (IDEA Report, p. 3) 


Adjusted progress rating on Item 23, Applications of course materials 
Predicted X93 = .27966X39+.43610X43-.003N-.1076Dy-. 1221 En+1.055086 
= .8679 +1.4587 -.102 -.0297 -.0206 +1.5509 = 3.7252 
Residual = X93 — Pred. X23 = 4.0690 -3.7252 = .3438 
Adjusted X23 = Grand Mean, Item 23 + (Residual)(1 + R’) 
= 3.9874 + (.3438)(1.2248) = 3.9874 + .4211 = 4.4085 (IDEA Report, p. 3) 


Adjusted progress rating on Item 31, Analysis and critical evaluation 
Predicted X3; = .13407X39+.42156X.3-.004N -.1995Dy-.1523En+ 1.96267 
= 4160 41.4100 -.136 -.0051 -.0256 +1.9627 = 3.5720 
Residual = X3; — Pred. X3; = 3.9310 — 3.5720 = .3590 
Adjusted X3; = Grand Mean, Item 31 + (Residual)(1 + R’) 
= 3.8438 + (.3590)(1.1186) = 3.8438 + .4016 = 4.2454 (IDEA Report, p. 3) 


Adjusted rating, Item 40—Increased positive attitude. 
Predicted X49 = .51242X39+.33205.X43-.001N-.2234Dn.+.0743En+1.00177 
=1.5902 +1.1106 -.034 -.0617 +.0125 +1.0018 = 3.6194 
Residual = X49 — Pred. X49 = 3.6552 — 3.6194 = .0358 
Adjusted X49 = Grand Mean, Item 40 + (Residual)(1 + R’) 
= 3.8611 + (.0358)(1.3606) = 3.8611 + .0487 = 3.9098 (IDEA Report, p. 2) 


Adjusted rating, Item 41—Excellence of teacher 
Predicted X4; = .24024X39+.23139X43-.001N-.1475Dn.-.1819EnN+2.58021 
= .7523 +.7740 -.034 -.0407 = -.0306 = +2.5802 = 4.0012 
Residual = X4; — Pred. X4; = 4.5517 — 4.0012 = .5505 
Adjusted X49 = Grand Mean, Item 41 + (Residual)(1 + R’) 
= 4.1815 + (.5505)(1.0883) = 4.1815 + 5991 = 4.7806 (IDEA Report, p. 2) 


Adjusted rating, Item 42—Excellence of course. 
Predicted X42 = .47249X39+.28732X43-.001N-.2141 Dy.+.0530En+1.35036 
= 1.4663 +0.9610 -.034 -.0591 +.0089 +1.3504 = 3.6935 
Residual = X42 — Pred. X42 = 3.8966 — 3.6935 = .2031 
Adjusted X42 = Grand Mean, Item 42 + (Residual)(1 + R’) 
= 3.9198 + (.2031)(1.2938) = 3.9198 + .2628 = 4.1826 (IDEA Report, p. 2) 
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IV. Calculating T Scores 


T Score = 50+[10(Obtained Mean-Grand Mean) divided by s.d.], where Grand Mean is 
National Mean and s.d is National standard deviation. Obtained mean for unadjusted T 
Score is the raw mean. The Obtained mean for the adjusted T Score is the adjusted mean 
calculated above. 


Item 21 
Unadjusted 
Adjusted 


Item 22 
Unadjusted 
Adjusted 


Item 23 
Unadjusted 
Adjusted 


Item 31 
Unadjusted 
Adjusted 


Item 40 
Unadjusted 
Adjusted 


Item 41 
Unadjusted 
Adjusted 


Item 42 
Unadjusted 
Adjusted 


PRO 


Mean Scores 
A=Obtained-Nat’l 


3.7241-4.0013= -.2772 
3.8593-4.0013= -.1420 


3.7241-3.9443= -.2202 
3.9002-3,9443= -.0441 


4.0690-3.9874= +.0816 
4.4085-3.9874= +.4211 


3.9310-3.8438= +.0874 
4.2454-3.8438= +.4016 


3.6552-3.8611= -.2059 
3.9098-3.8611= +.0487 


4.5517-4.1815= +.3702 
4.7806-4.1815= +.5991 


3.8966-3.9198= -.0232 
4.1826-3.9198= +.2628 


B=Nat’l 


s. d. 
494 
494 


485 
A485 


516 
516 


589 
589 


.602 
.602 


642 
642 


.607 
.607 


10(A/B) - IDEA Report Page 
-5.61 (+50 = 44, p. 3) 
-2.87 (+50 = 47, p. 3) 


-4.54 (+50 = 45, p. 3) 
-0.91 (+50 = 49, p. 3) 


1.58 (+50 = 52, p. 3) 
8.16 (+50 = 58, p. 3) 


1.48 (+50 =51, p. 3) 
6.82 (+50 = 57, p. 3) 


-3.42 (+50 = 47, p. 2) 
0.81 (+50 =51, p. 2) 


5.77 (+50 = 56, p. 2) 
9.53 (+50 = 60, p. 2) 


0.38 (+50 = 50, p. 2) 
4.53 (+50 =55, p. 2) 


Unadjusted 2(44 +45) + (52 +51) = 178 + 103 = 281 divided by 6 = 46.8 (p. 2) 
Adjusted 2(47 + 49) + (58 + 57) = 192 + 115 = 307 divided by 6 = 51.2 (p. 2) 
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Appendix C 


Regression Coefficients and Constants for 
Adjusting Ratings on the Revised Short Form 
Effective October 1, 2002 
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Adj usted CM, 5 WH 13 #Enroll * pZ Grand 
| Wtem21_ | 1.7559 _| 0.2572 | 03842 | 0 | 1.1737 | 4.0013 _| 
| Item22_ | 1.7619 | 0.2273 | 03041 | 0 | 1.1593 | 3.9443 _ | 
| Item25__| 1.6622 | 0.1700 | 04742 | oo | tio | 3.9285_| 
| Item30__| 1.4258 | 0.2189 | 04502 | 0 | t.1088_ | 3.7779 _| 


CM5=Course Motivation — Short Form Item 15. / really wanted to take this course regardless of who taught it. 

WH)3=Student Work Habits — Short Form Item 13. Asa rule, I put forth more effort than other students on 
academic work . 

#Enrolled=Number of students enrolled in the course as indicated by the instructor on the Faculty Information 
Form. 

Note: Analyses are based on a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 


: 9 
: 2 
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